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CHAPTER 1 
ABOUT THIS MANUAL 


The Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A: System Programming Guide, 
Part 1 (order number 253668), the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3B: 
System Programming Guide, Part 2 (order number 253669), the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 3C: System Programming Guide, Part 3 (order number 326019), and the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 3D:System Programming Guide, Part 4 (order 
number 332831) are part of a set that describes the architecture and programming environment of Intel 64 and IA- 
32 Architecture processors. The other volumes in this set are: 


° Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture (order number 
253665). 


° = =Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volumes 2A, 2B, 2C & 2D: Instruction Set 
Reference (order numbers 253666, 253667, 326018 and 334569). 


° The Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4: Model-Specific Registers 
(order number 335592). 


The Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, describes the basic architecture 
and programming environment of Intel 64 and IA-32 processors. The Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volumes 2A, 2B, 2C & 2D, describe the instruction set of the processor and the opcode struc- 
ture. These volumes apply to application programmers and to programmers who write operating systems or exec- 
utives. The Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volumes 3A, 3B, 3C & 3D, describe 
the operating-system support environment of Intel 64 and IA-32 processors. These volumes target operating- 
system and BIOS designers. In addition, Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
3B, and Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C address the programming 
environment for classes of software that host operating systems. The Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 4, describes the model-specific registers of Intel 64 and IA-32 processors. 


1.1 INTEL® 64 AND IA-32 PROCESSORS COVERED IN THIS MANUAL 


This manual set includes information pertaining primarily to the most recent Intel 64 and IA-32 processors, which 
include: 


* Pentium® processors 

° P6 family processors 

* Pentium® 4 processors 

* Pentium® M processors 

* Intel® Xeon® processors 

* Pentium® D processors 

* Pentium® processor Extreme Editions 

* 64-bit Intel® Xeon® processors 

* Intel® Core™ Duo processor 

* Intel® Core™ Solo processor 

* Dual-Core Intel® Xeon® processor LV 

* Intel® Core™2 Duo processor 

* Intel® Core™2 Quad processor Q6000 series 
* Intel® xeon® processor 3000, 3200 series 
* Intel® Xeon® processor 5000 series 

* Intel® xeon® processor 5100, 5300 series 
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* Intel® Core™2 Extreme processor X7000 and X6800 series 
* Intel® Core™2 Extreme QX6000 series 

* Intel® Xeon® processor 7100 series 

* Intel® Pentium® Dual-Core processor 

* Intel® Xeon® processor 7200, 7300 series 

* Intel® Core™2 Extreme QX9000 series 

* Intel® Xeon® processor 5200, 5400, 7400 series 

* Intel® Core™2 Extreme processor QX9000 and X9000 series 
* Intel® Core™2 Quad processor Q9000 series 

* Intel® Core™2 Duo processor E8000, T9000 series 


* Intel® Atom™ processors 200, 300, D400, D500, D2000, N200, N400, N2000, E2000, Z500, Z600, Z2000, 
C1000 series are built from 45 nm and 32 nm processes. 


* Intel® Core™ i7 processor 

* Intel® Core™ i5 processor 

* Intel® Xeon® processor E7-8800/4800/2800 product families 
* Intel® Core™ i7-3930K processor 

* 2nd generation Intel® Core™ i7-2xxx, Intel® Core™ i5-2xxx, Intel® Core™ i3-2xxx processor series 
* Intel® Xeon® processor E3-1200 product family 

* Intel® Xeon® processor E5-2400/1400 product family 

* Intel® Xeon® processor E5-4600/2600/1600 product family 
* 3rd generation Intel® Core™ processors 

* Intel® Xeon® processor E3-1200 v2 product family 

* Intel® Xeon® processor E5-2400/1400 v2 product families 

* Intel® Xeon® processor E5-4600/2600/1600 v2 product families 
* Intel® Xeon® processor E7-8800/4800/2800 v2 product families 
* 4th generation Intel® Core™ processors 

* The Intel® Core™ M processor family 

* Intel® Core™ i7-59xx Processor Extreme Edition 

* Intel® Core™ i7-49xx Processor Extreme Edition 

* Intel® Xeon® processor E3-1200 v3 product family 

* Intel® Xeon® processor E5-2600/1600 v3 product families 

* 5th generation Intel® Core™ processors 

* Intel® Xeon® processor D-1500 product family 

* Intel® Xeon® processor E5 v4 family 

* Intel® Atom™ processor X7-Z8000 and X5-Z8000 series 

* Intel® Atom™ processor Z3400 series 

* Intel® Atom™ processor Z3500 series 

* 6th generation Intel® Core™ processors 

* Intel® Xeon® processor E3-1500m v5 product family 

* 7th generation Intel® Core™ processors 

* Intel® Xeon Phi™ Processor 3200, 5200, 7200 Series 

* Intel® Xeon® Processor Scalable Family 

* 8th generation Intel® Core™ processors 

* Intel® Xeon Phi™ Processor 7215, 7285, 7295 Series 
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* Intel® Xeon® E processors 

* 9th generation Intel® Core™ processors 

* 2nd generation Intel® Xeon® Processor Scalable Family 
* 10th generation Intel® Core™ processors 

* 11th generation Intel® Core™ processors 

* 3rd generation Intel® Xeon® Processor Scalable Family 


P6 family processors are IA-32 processors based on the P6 family microarchitecture. This includes the Pentium® 
Pro, Pentium® II, Pentium® III, and Pentium® III Xeon® processors. 


The Pentium® 4, Pentium® D, and Pentium® processor Extreme Editions are based on the Intel NetBurst® micro- 
architecture. Most early Intel® Xeon® processors are based on the Intel NetBurst® microarchitecture. Intel Xeon 
processor 5000, 7100 series are based on the Intel NetBurst® microarchitecture. 


The Intel® Core™ Duo, Intel® Core™ Solo and dual-core Intel® Xeon® processor LV are based on an improved 
Pentium® M processor microarchitecture. 


The Intel® Xeon® processor 3000, 3200, 5100, 5300, 7200, and 7300 series, Intel® Pentium® dual-core, Intel® 
Core™2 Duo, Intel® Core™2 Quad, and Intel® Core™2 Extreme processors are based on Intel® Core™ microarchi- 
tecture. 


The Intel® Xeon® processor 5200, 5400, 7400 series, Intel® Core™2 Quad processor Q9000 series, and Intel® 
Core™2 Extreme processors QX9000, X9000 series, Intel® Core™2 processor E8000 series are based on Enhanced 
Intel® Core™ microarchitecture. 


The Intel® Atom™ processors 200, 300, D400, D500, D2000, N200, N400, N2000, E2000, Z500, Z600, Z2000, 
C1000 series are based on the Intel® Atom™ microarchitecture and supports Intel 64 architecture. 


P6 family, Pentium® M, Intel® Core™ Solo, Intel® Core™ Duo processors, dual-core Intel® Xeon® processor LV, 
and early generations of Pentium 4 and Intel Xeon processors support IA-32 architecture. The Intel® Atom™ 
processor Z5xx series support IA-32 architecture. 


The Intel® Xeon® processor 3000, 3200, 5000, 5100, 5200, 5300, 5400, 7100, 7200, 7300, 7400 series, Intel® 
Core™ 2 Duo, Intel® Core™2 Extreme, Intel® Core™2 Quad processors, Pentium® D processors, Pentium® Dual- 
Core processor, newer generations of Pentium 4 and Intel Xeon processor family support Intel® 64 architecture. 


The Intel® Core™ i7 processor and Intel® Xeon® processor 3400, 5500, 7500 series are based on 45 nm Nehalem 
microarchitecture. Westmere microarchitecture is a 32 nm version of the Nehalem microarchitecture. Intel® 
Xeon® processor 5600 series, Intel Xeon processor E7 and various Intel Core i7, i5, i3 processors are based on the 
Westmere microarchitecture. These processors support Intel 64 architecture. 


The Intel® Xeon® processor E5 family, Intel® Xeon® processor E3-1200 family, Intel® Xeon® processor E7- 
8800/4800/2800 product families, Intel® Core™ i7-3930K processor, and 2nd generation Intel® Core™ i7-2xxx, 
Intel® Core™ i5-2xxx, Intel® Core™ i3-2xxx processor series are based on the Sandy Bridge microarchitecture and 
support Intel 64 architecture. 


The Intel® Xeon® processor E7-8800/4800/2800 v2 product families, Intel® Xeon® processor E3-1200 v2 product 
family and 3rd generation Intel® Core™ processors are based on the Ivy Bridge microarchitecture and support 
Intel 64 architecture. 


The Intel® Xeon® processor E5-4600/2600/1600 v2 product families, Intel® Xeon® processor E5-2400/1400 v2 
product families and Intel® Core™ i7-49xx Processor Extreme Edition are based on the Ivy Bridge-E microarchitec- 
ture and support Intel 64 architecture. 


The Intel® Xeon® processor E3-1200 v3 product family and 4th Generation Intel® Core™ processors are based on 
the Haswell microarchitecture and support Intel 64 architecture. 


The Intel® Xeon® processor E5-2600/1600 v3 product families and the Intel® Core™ i7-59xx Processor Extreme 
Edition are based on the Haswell-E microarchitecture and support Intel 64 architecture. 


The Intel® Atom™ processor Z8000 series is based on the Airmont microarchitecture. 


The Intel® Atom™ processor Z3400 series and the Intel® Atom™ processor Z3500 series are based on the Silver- 
mont microarchitecture. 
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The Intel® Core™ M processor family, 5th generation Intel® Core™ processors, Intel® Xeon® processor D-1500 
product family and the Intel® Xeon® processor E5 v4 family are based on the Broadwell microarchitecture and 
support Intel 64 architecture. 


The Intel® Xeon® Processor Scalable Family, Intel® Xeon® processor E3-1500m v5 product family and 6th gener- 
ation Intel® Core™ processors are based on the Skylake microarchitecture and support Intel 64 architecture. 


The 7th generation Intel® Core™ processors are based on the Kaby Lake microarchitecture and support Intel 64 
architecture. 


The Intel® Atom™ processor C series, the Intel® Atom™ processor X series, the Intel® Pentium® processor J 
series, the Intel® Celeron® processor J series, and the Intel® Celeron® processor N series are based on the Gold- 
mont microarchitecture. 


The Intel® Xeon Phi™ Processor 3200, 5200, 7200 Series is based on the Knights Landing microarchitecture and 
supports Intel 64 architecture. 


The Intel® Pentium® Silver processor series, the Intel® Celeron® processor J series, and the Intel® Celeron® 
processor N series are based on the Goldmont Plus microarchitecture. 


The 8th generation Intel® Core™ processors, 9th generation Intel® Core™ processors, and Intel® Xeon® E proces- 
sors are based on the Coffee Lake microarchitecture and support Intel 64 architecture. 


The Intel® Xeon Phi™ Processor 7215, 7285, 7295 Series is based on the Knights Mill microarchitecture and 
supports Intel 64 architecture. 


The 2nd generation Intel® Xeon® Processor Scalable Family is based on the Cascade Lake product and supports 
Intel 64 architecture. 


Some 10th generation Intel® Core™ processors are based on the Ice Lake microarchitecture, and some are based 
on the Comet Lake microarchitecture; both support Intel 64 architecture. 


Some 11th generation Intel® Core™ processors are based on the Tiger Lake microarchitecture, and some are 
based on the Rocket Lake microarchitecture; both support Intel 64 architecture. 


Some 3rd generation Intel® Xeon® Processor Scalable Family processors are based on the Cooper Lake product, 
and some are based on the Ice Lake microarchitecture; both support Intel 64 architecture. 


IA-32 architecture is the instruction set architecture and programming environment for Intel's 32-bit microproces- 
sors. Intel® 64 architecture is the instruction set architecture and programming environment which is the superset 
of Intel’s 32-bit and 64-bit architectures. It is compatible with the IA-32 architecture. 


Lé OVERVIEW OF THE SYSTEM PROGRAMMING GUIDE 


A description of this manual’s content follows!?: 


Chapter 1 — About This Manual. Gives an overview of all eight volumes of the Inte/® 64 and IA-32 Architectures 
Software Developer’s Manual. It also describes the notational conventions in these manuals and lists related Intel 
manuals and documentation of interest to programmers and hardware designers. 


Chapter 2 — System Architecture Overview. Describes the modes of operation used by Intel 64 and IA-32 
processors and the mechanisms provided by the architectures to support operating systems and executives, 
including the system-oriented registers and data structures and the system-oriented instructions. The steps neces- 
sary for switching between real-address and protected modes are also identified. 


Chapter 3 — Protected-Mode Memory Management. Describes the data structures, registers, and instructions 
that support segmentation and paging. The chapter explains how they can be used to implement a “flat” (unseg- 
mented) memory model or a segmented memory model. 


Chapter 4 — Paging. Describes the paging modes supported by Intel 64 and IA-32 processors. 


1. Model-Specific Registers have been moved out of this volume and into a separate volume: Intel® 64 and IA-32 Architectures Soft- 
ware Developer's Manual, Volume 4. 
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Chapter 5 — Protection. Describes the support for page and segment protection provided in the Intel 64 and IA- 
32 architectures. This chapter also explains the implementation of privilege rules, stack switching, pointer valida- 
tion, user and supervisor modes. 


Chapter 6 — Interrupt and Exception Handling. Describes the basic interrupt mechanisms defined in the Intel 
64 and IA-32 architectures, shows how interrupts and exceptions relate to protection, and describes how the archi- 
tecture handles each exception type. Reference information for each exception is given in this chapter. Includes 
programming the LINTO and LINT1 inputs and gives an example of how to program the LINTO and LINT1 pins for 
specific interrupt vectors. 


Chapter 7 — Task Management. Describes mechanisms the Intel 64 and IA-32 architectures provide to support 
multitasking and inter-task protection. 


Chapter 8 — Multiple-Processor Management. Describes the instructions and flags that support multiple 
processors with shared memory, memory ordering, and Intel® Hyper-Threading Technology. Includes MP initializa- 
tion for P6 family processors and gives an example of how to use the MP protocol to boot P6 family processors in 
an MP system. 


Chapter 9 — Processor Management and Initialization. Defines the state of an Intel 64 or IA-32 processor 
after reset initialization. This chapter also explains how to set up an Intel 64 or IA-32 processor for real-address 
mode operation and protected- mode operation, and how to switch between modes. 


Chapter 10 — Advanced Programmable Interrupt Controller (APIC). Describes the programming interface 
to the local APIC and gives an overview of the interface between the local APIC and the I/O APIC. Includes APIC bus 
message formats and describes the message formats for messages transmitted on the APIC bus for P6 family and 
Pentium processors. 


Chapter 11 — Memory Cache Control. Describes the general concept of caching and the caching mechanisms 
supported by the Intel 64 or IA-32 architectures. This chapter also describes the memory type range registers 
(MTRRs) and how they can be used to map memory types of physical memory. Information on using the new cache 
control and memory streaming instructions introduced with the Pentium III, Pentium 4, and Intel Xeon processors 
is also given. 


Chapter 12 — Intel® MMX™ Technology System Programming. Describes those aspects of the Intel® MMX™ 
technology that must be handled and considered at the system programming level, including: task switching, 
exception handling, and compatibility with existing system environments. 


Chapter 13 — System Programming For Instruction Set Extensions And Processor Extended States. 
Describes the operating system requirements to support SSE/SSE2/SSE3/SSSE3/SSE4 extensions, including task 
switching, exception handling, and compatibility with existing system environments. The latter part of this chapter 
describes the extensible framework of operating system requirements to support processor extended states. 
Processor extended state may be required by instruction set extensions beyond those of 
SSE/SSE2/SSE3/SSSE3/SSE4 extensions. 


Chapter 14 — Power and Thermal Management. Describes facilities of Intel 64 and IA-32 architecture used for 
power management and thermal monitoring. 


Chapter 15 — Machine-Check Architecture. Describes the machine-check architecture and machine-check 
exception mechanism found in the Pentium 4, Intel Xeon, and P6 family processors. Additionally, a signaling mech- 
anism for software to respond to hardware corrected machine check error is covered. 


Chapter 16 — Interpreting Machine-Check Error Codes. Gives an example of how to interpret the error codes 
for a machine-check error that occurred on a P6 family processor. 


Chapter 17 — Debug, Branch Profile, TSC, and Resource Monitoring Features. Describes the debugging 
registers and other debug mechanism provided in Intel 64 or IA-32 processors. This chapter also describes the 
time-stamp counter. 


Chapter 18 — Performance Monitoring. Describes the Intel 64 and IA-32 architectures’ facilities for monitoring 
performance. 


Chapter 19 — 8086 Emulation. Describes the real-address and virtual-8086 modes of the IA-32 architecture. 


Chapter 20 — Mixing 16-Bit and 32-Bit Code. Describes how to mix 16-bit and 32-bit code modules within the 
same program or task. 


Chapter 21 — IA-32 Architecture Compatibility. Describes architectural compatibility among IA-32 proces- 
sors. 
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Chapter 22 — Introduction to Virtual Machine Extensions. Describes the basic elements of virtual machine 
architecture and the virtual machine extensions for Intel 64 and IA-32 Architectures. 


Chapter 23 — Virtual Machine Control Structures. Describes components that manage VMX operation. These 
include the working-VMCS pointer and the controlling-VMCS pointer. 


Chapter 24 — VMX Non-Root Operation. Describes the operation of a VMX non-root operation. Processor oper- 
ation in VMX non-root mode can be restricted programmatically such that certain operations, events or conditions 
can cause the processor to transfer control from the guest (running in VMX non-root mode) to the monitor software 
(running in VMX root mode). 


Chapter 25 — VM Entries. Describes VM entries. VM entry transitions the processor from the VMM running in VMX 
root-mode to a VM running in VMX non-root mode. VM-Entry is performed by the execution of VMLAUNCH or VMRE- 
SUME instructions. 


Chapter 26 — VM Exits. Describes VM exits. Certain events, operations or situations while the processor is in VMX 
non-root operation may cause VM-exit transitions. In addition, VM exits can also occur on failed VM entries. 


Chapter 27 — VMX Support for Address Translation. Describes virtual-machine extensions that support 
address translation and the virtualization of physical memory. 


Chapter 28 — APIC Virtualization and Virtual Interrupts. Describes the VMCS including controls that enable 
the virtualization of interrupts and the Advanced Programmable Interrupt Controller (APIC). 


Chapter 29 — VMX Instruction Reference. Describes the virtual-machine extensions (VMX). VMX is intended 
for a system executive to support virtualization of processor hardware and a system software layer acting as a host 
to multiple guest software environments. 


Chapter 30 — System Management Mode. Describes Intel 64 and IA-32 architectures’ system management 
mode (SMM) facilities. 


Chapter 31 — Intel® Processor Trace. Describes details of Intel® Processor Trace. 


Chapter 32 — Introduction to Intel® Software Guard Extensions. Provides an overview of the Intel® Soft- 
ware Guard Extensions (Intel® SGX) set of instructions. 


Chapter 33 — Enclave Access Control and Data Structures. Describes Enclave Access Control procedures and 
defines various Intel SGX data structures. 


Chapter 34 — Enclave Operation. Describes enclave creation and initialization, adding pages and measuring an 
enclave, and enclave entry and exit. 


Chapter 35 — Enclave Exiting Events. Describes enclave-exiting events (EEE) and asynchronous enclave exit 
(AEX). 


Chapter 36 — SGX Instruction References. Describes the supervisor and user level instructions provided by 
Intel SGX. 


Chapter 37 — Intel® SGX Interactions with IA32 and Intel® 64 Architecture. Describes the Intel SGX 
collection of enclave instructions for creating protected execution environments on processors supporting IA32 and 
Intel 64 architectures. 


Chapter 38 — Enclave Code Debug and Profiling. Describes enclave code debug processes and options. 


Appendix A — VMX Capability Reporting Facility. Describes the VMX capability MSRs. Support for specific VMX 
features is determined by reading capability MSRs. 


Appendix B — Field Encoding in VMCS. Enumerates all fields in the VMCS and their encodings. Fields are 
grouped by width (16-bit, 32-bit, etc.) and type (guest-state, host-state, etc.). 


Appendix C — VM Basic Exit Reasons. Describes the 32-bit fields that encode reasons for a VM exit. Examples 
of exit reasons include, but are not limited to: software interrupts, processor exceptions, software traps, NMIs, 
external interrupts, and triple faults. 


1.3 NOTATIONAL CONVENTIONS 


This manual uses specific notation for data-structure formats, for symbolic representation of instructions, and for 
hexadecimal and binary numbers. A review of this notation makes the manual easier to read. 
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1.3.1 Bit and Byte Order 


In illustrations of data structures in memory, smaller addresses appear toward the bottom of the figure; addresses 
increase toward the top. Bit positions are numbered from right to left. The numerical value of a set bit is equal to 

two raised to the power of the bit position. Intel 64 and IA-32 processors are “little endian” machines; this means 
the bytes of a word are numbered starting from the least significant byte. Figure 1-1 illustrates these conventions. 


1.3.2 Reserved Bits and Software Compatibility 


In many register and memory layout descriptions, certain bits are marked as reserved. When bits are marked as 

reserved, it is essential for compatibility with future processors that software treat these bits as having a future, 

though unknown, effect. The behavior of reserved bits should be regarded as not only undefined, but unpredict- 

able. Software should follow these guidelines in dealing with reserved bits: 

* Do not depend on the states of any reserved bits when testing the values of registers which contain such bits. 
Mask out the reserved bits before testing. 


* Do not depend on the states of any reserved bits when storing to memory or to a register. 
* Do not depend on the ability to retain information written into any reserved bits. 


¢ When loading a register, always load the reserved bits with the values indicated in the documentation, if any, 
or reload them with values previously read from the same register. 


NOTE 


Avoid any software dependence upon the state of reserved bits in Intel 64 and IA-32 registers. 
Depending upon the values of reserved register bits will make software dependent upon the 
unspecified manner in which the processor handles these bits. Programs that depend upon 
reserved values risk incompatibility with future processors. 


Data Structure 


Highest ; 
Address 32 24 23 16 15 87 0 <«— Bit offset 


Lowest 
Address 


Byte Offset 


Figure 1-1. Bit and Byte Order 
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1.33 Instruction Operands 


When instructions are represented symbolically, a subset of assembly language is used. In this subset, an instruc- 
tion has the following format: 
label: mnemonic argument1, argument2, argument3 
where: 
¢ A label is an identifier which is followed by a colon. 
¢ Amnemonic is a reserved name for a class of instruction opcodes which have the same function. 


¢ The operands argumenti, argument2, and argument3 are optional. There may be from zero to three 
operands, depending on the opcode. When present, they take the form of either literals or identifiers for data 
items. Operand identifiers are either reserved names of registers or are assumed to be assigned to data items 
declared in another part of the program (which may not be shown in the example). 


When two operands are present in an arithmetic or logical instruction, the right operand is the source and the left 
operand is the destination. 


For example: 


LOADREG: MOV EAX, SUBTOTAL 


In this example LOADREG is a label, MOV is the mnemonic identifier of an opcode, EAX is the destination operand, 
and SUBTOTAL is the source operand. Some assembly languages put the source and destination in reverse order. 


1.3.4 Hexadecimal and Binary Numbers 


Base 16 (hexadecimal) numbers are represented by a string of hexadecimal digits followed by the character H (for 
example, F82EH). A hexadecimal digit is a character from the following set: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, 
E, and F. 


Base 2 (binary) numbers are represented by a string of 1s and Os, sometimes followed by the character B (for 
example, 1010B). The “B” designation is only used in situations where confusion as to the type of number might 
arise. 


1.3.5 Segmented Addressing 


The processor uses byte addressing. This means memory is organized and accessed as a sequence of bytes. 
Whether one or more bytes are being accessed, a byte address is used to locate the byte or bytes memory. The 
range of memory that can be addressed is called an address space. 


The processor also supports segmented addressing. This is a form of addressing where a program may have many 
independent address spaces, called segments. For example, a program can keep its code (instructions) and stack 
in separate segments. Code addresses would always refer to the code space, and stack addresses would always 
refer to the stack space. The following notation is used to specify a byte address within a segment: 


Segment-register: Byte-address 


For example, the following segment address identifies the byte at address FF79H in the segment pointed by the DS 
register: 


DS:FF79H 


The following segment address identifies an instruction address in the code segment. The CS register points to the 
code segment and the EIP register contains the address of the instruction. 


CS:EIP 
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1.3.6 Syntax for CPUID, CR, and MSR Values 


Obtain feature flags, status, and system information by using the CPUID instruction, by checking control register 
bits, and by reading model-specific registers. We are moving toward a single syntax to represent this type of infor- 
mation. See Figure 1-2. 


CPUID Input and Output 


CPUID.01H:EDX.SSE[bit 25] = 1 


Input value for EAX register 


Output register and feature flag or field 
name with bit position(s) 


Value (or range) of output 


Control Register Values 


CR4.OSFXSR)Ibit 9] = 1 


Example CR name 


Feature flag or field name 
with bit position(s) 


Value (or range) of output 


Model-Specific Register Values 


IA32_MISC_ENABLE.ENABLEFOPCODEDbit 2] = 1 
) 


Example MSR name 


Feature flag or field name with bit position(s) 


Value (or range) of output 


Figure 1-2. Syntax for CPUID, CR, and MSR Data Presentation 


1.3.7 Exceptions 


An exception is an event that typically occurs when an instruction causes an error. For example, an attempt to 
divide by zero generates an exception. However, some exceptions, such as breakpoints, occur under other condi- 
tions. Some types of exceptions may provide error codes. An error code reports additional information about the 
error. An example of the notation used to show an exception and error code is shown below: 


#PF(fault code) 


This example refers to a page-fault exception under conditions where an error code naming a type of fault is 
reported. Under some conditions, exceptions which produce error codes may not be able to report an accurate 
code. In this case, the error code is zero, as shown below for a general-protection exception: 


#GP(0) 
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1.4 RELATED LITERATURE 


Literature related to Intel 64 and IA-32 processors is listed and viewable on-line at: 


https://software.intel.com/en-us/articles/intel-sdm 


See also: 


The latest security information on Intel® products: 
https://www.intel.com/content/www/us/en/security-center/default.html 


Software developer resources, guidance and insights for security advisories: 
https://software.intel.com/security-software-guidance/ 


The data sheet for a particular Intel 64 or IA-32 processor 
The specification update for a particular Intel 64 or IA-32 processor 


Intel® C++ Compiler documentation and online help: 
http://software.intel.com/en-us/articles/intel-compilers/ 


Intel® Fortran Compiler documentation and online help: 
http ://software.intel.com/en-us/articles/intel-compilers/ 


Intel® Software Development Tools: 
https://software.intel.com/en-us/intel-sdp-home 


Intel® 64 and IA-32 Architectures Software Developer’s Manual (in one, four or ten volumes): 
https://software.intel.com/en-us/articles/intel-sdm 


Intel® 64 and IA-32 Architectures Optimization Reference Manual: 
https://software.intel.com/en-us/articles/intel-sdm#optimization 


Intel® Trusted Execution Technology Measured Launched Environment Programming Guide: 
http://www. intel.com/content/www/us/en/software-developers/intel-txt-software-development-guide.html 


Intel® Software Guard Extensions (Intel® SGX) Information 
https://software.intel.com/en-us/isa-extensions/intel-sgx 


Developing Multi-threaded Applications: A Platform Consistent Approach: 
https://software.intel.com/sites/default/files/article/147714/51534-developing-multithreaded-applica- 
tions. pdf 


Using Spin-Loops on Intel® Pentium® 4 Processor and Intel® Xeon® Processor: 
https://software.intel.com/sites/default/files/22/30/25602 


Performance Monitoring Unit Sharing Guide 
http://software.intel.com/file/30388 


Literature related to select features in future Intel processors are available at: 


Intel® Architecture Instruction Set Extensions Programming Reference 
https://software.intel.com/en-us/isa-extensions 


More relevant links are: 


Intel® Developer Zone: 

https://software.intel.com/en-us 

Developer centers: 

http://www. intel.com/content/www/us/en/hardware-developers/developer-centers.html 
Processor support general link: 

http://www.intel.com/support/processors/ 

Intel® Hyper-Threading Technology (Intel® HT Technology): 

http://www. intel.com/technology/platform-technology/hyper-threading/index.htm 
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CHAPTER 2 
SYSTEM ARCHITECTURE OVERVIEW 


IA-32 architecture (beginning with the Intel386 processor family) provides extensive support for operating-system 
and system-development software. This support offers multiple modes of operation, which include: 


° Real mode, protected mode, virtual 8086 mode, and system management mode. These are sometimes 
referred to as legacy modes. 


Intel 64 architecture supports almost all the system programming facilities available in IA-32 architecture and 
extends them to a new operating mode (IA-32e mode) that supports a 64-bit programming environment. IA-32e 
mode allows software to operate in one of two sub-modes: 


° 64-bit mode supports 64-bit OS and 64-bit applications 

* Compatibility mode allows most legacy software to run; it co-exists with 64-bit applications under a 64-bit OS. 
The IA-32 system-level architecture includes features to assist in the following operations: 
¢ Memory management 

¢ Protection of software modules 

¢ Multitasking 

* Exception and interrupt handling 

* Multiprocessing 

* Cache management 

¢ Hardware resource and power management 

¢ Debugging and performance monitoring 


This chapter provides a description of each part of this architecture. It also describes the system registers that are 
used to set up and control the processor at the system level and gives a brief overview of the processor’s system- 
level (operating system) instructions. 


Many features of the system-level architecture are used only by system programmers. However, application 
programmers may need to read this chapter and the following chapters in order to create a reliable and secure 
environment for application programs. 


This overview and most subsequent chapters of this book focus on protected-mode operation of the IA-32 architec- 
ture. IA-32e mode operation of the Intel 64 architecture, as it differs from protected mode operation, is also 
described. 


All Intel 64 and IA-32 processors enter real-address mode following a power-up or reset (See Chapter 9, “Processor 
Management and Initialization”). Software then initiates the switch from real-address mode to protected mode. If 
IA-32e mode operation is desired, software also initiates a switch from protected mode to IA-32e mode. 


Z.1 OVERVIEW OF THE SYSTEM-LEVEL ARCHITECTURE 


System-level architecture consists of a set of registers, data structures, and instructions designed to support basic 
system-level operations such as memory management, interrupt and exception handling, task management, and 
control of multiple processors. 


Figure 2-1 provides a summary of system registers and data structures that applies to 32-bit modes. System regis- 
ters and data structures that apply to IA-32e mode are shown in Figure 2-2. 
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Figure 2-2. System-Level Registers and Data Structures in |[A-32e Mode and 4-Level Paging 


2.1.1 Global and Local Descriptor Tables 


When operating in protected mode, all memory accesses pass through either the global descriptor table (GDT) or 
an optional local descriptor table (LDT) as shown in Figure 2-1. These tables contain entries called segment 


descriptors. Segment descriptors provide the base address of segments well as access rights, type, and usage 
information. 
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Each segment descriptor has an associated segment selector. A segment selector provides the software that uses 
it with an index into the GDT or LDT (the offset of its associated segment descriptor), a global/local flag (deter- 
mines whether the selector points to the GDT or the LDT), and access rights information. 


To access a byte in a segment, a segment selector and an offset must be supplied. The segment selector provides 
access to the segment descriptor for the segment (in the GDT or LDT). From the segment descriptor, the processor 
obtains the base address of the segment in the linear address space. The offset then provides the location of the 
byte relative to the base address. This mechanism can be used to access any valid code, data, or stack segment, 
provided the segment is accessible from the current privilege level (CPL) at which the processor is operating. The 
CPL is defined as the protection level of the currently executing code segment. 


See Figure 2-1. The solid arrows in the figure indicate a linear address, dashed lines indicate a segment selector, 
and the dotted arrows indicate a physical address. For simplicity, many of the segment selectors are shown as 
direct pointers to a segment. However, the actual path from a segment selector to its associated segment is always 
through a GDT or LDT. 


The linear address of the base of the GDT is contained in the GDT register (GDTR); the linear address of the LDT is 
contained in the LDT register (LDTR). 


2.1.1.1 Global and Local Descriptor Tables in |A-32e Mode 


GDTR and LDTR registers are expanded to 64-bits wide in both IA-32e sub-modes (64-bit mode and compatibility 
mode). For more information: see Section 3.5.2, “Segment Descriptor Tables in IA-32e Mode.” 


Global and local descriptor tables are expanded in 64-bit mode to support 64-bit base addresses, (16-byte LDT 
descriptors hold a 64-bit base address and various attributes). In compatibility mode, descriptors are not 
expanded. 


2lve System Segments, Segment Descriptors, and Gates 


Besides code, data, and stack segments that make up the execution environment of a program or procedure, the 
architecture defines two system segments: the task-state segment (TSS) and the LDT. The GDT is not considered 
a segment because it is not accessed by means of a segment selector and segment descriptor. TSSs and LDTs have 
segment descriptors defined for them. 


The architecture also defines a set of special descriptors called gates (call gates, interrupt gates, trap gates, and 
task gates). These provide protected gateways to system procedures and handlers that may operate at a different 
privilege level than application programs and most procedures. For example, a CALL to a call gate can provide 
access to a procedure in a code segment that is at the same or a numerically lower privilege level (more privileged) 
than the current code segment. To access a procedure through a call gate, the calling procedure! supplies the 
selector for the call gate. The processor then performs an access rights check on the call gate, comparing the CPL 
with the privilege level of the call gate and the destination code segment pointed to by the call gate. 


If access to the destination code segment is allowed, the processor gets the segment selector for the destination 
code segment and an offset into that code segment from the call gate. If the call requires a change in privilege 
level, the processor also switches to the stack for the targeted privilege level. The segment selector for the new 
stack is obtained from the TSS for the currently running task. Gates also facilitate transitions between 16-bit and 
32-bit code segments, and vice versa. 


2.1.2.1 Gates in IA-32e Mode 


In IA-32e mode, the following descriptors are 16-byte descriptors (expanded to allow a 64-bit base): LDT descrip- 
tors, 64-bit TSSs, call gates, interrupt gates, and trap gates. 


Call gates facilitate transitions between 64-bit mode and compatibility mode. Task gates are not supported in IA- 
32e mode. On privilege level changes, stack segment selectors are not read from the TSS. Instead, they are set to 
NULL. 


1. The word “procedure” is commonly used in this document as a general term for a logical unit or block of code (such as a program, pro- 
cedure, function, or routine). 
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rae is) Task-State Segments and Task Gates 


The TSS (see Figure 2-1) defines the state of the execution environment for a task. It includes the state of general- 
purpose registers, segment registers, the EFLAGS register, the EIP register, and segment selectors with stack 
pointers for three stack segments (one stack for each privilege level). The TSS also includes the segment selector 
for the LDT associated with the task and the base address of the paging-structure hierarchy. 


All program execution in protected mode happens within the context of a task (called the current task). The 
segment selector for the TSS for the current task is stored in the task register. The simplest method for switching 
to a task is to make a call or jump to the new task. Here, the segment selector for the TSS of the new task is given 
in the CALL or JMP instruction. In switching tasks, the processor performs the following actions: 


Stores the state of the current task in the current TSS. 
Loads the task register with the segment selector for the new task. 
Accesses the new TSS through a segment descriptor in the GDT. 


h WN 


Loads the state of the new task from the new TSS into the general-purpose registers, the segment registers, 
the LDTR, control register CR3 (base address of the paging-structure hierarchy), the EFLAGS register, and the 
EIP register. 


5. Begins execution of the new task. 


A task can also be accessed through a task gate. A task gate is similar to a call gate, except that it provides access 
(through a segment selector) to a TSS rather than a code segment. 


2.1.3.1 Task-State Segments in IA-32e Mode 


Hardware task switches are not supported in IA-32e mode. However, TSSs continue to exist. The base address of 
a TSS is specified by its descriptor. 


A 64-bit TSS holds the following information that is important to 64-bit operation: 
¢ Stack pointer addresses for each privilege level 

¢ Pointer addresses for the interrupt stack table 

° Offset address of the 10-permission bitmap (from the TSS base) 


The task register is expanded to hold 64-bit base addresses in IA-32e mode. See also: Section 7.7, “Task Manage- 
ment in 64-bit Mode.” 


2.1.4 Interrupt and Exception Handling 


External interrupts, software interrupts and exceptions are handled through the interrupt descriptor table (IDT). 
The IDT stores a collection of gate descriptors that provide access to interrupt and exception handlers. Like the 
GDT, the IDT is not a segment. The linear address for the base of the IDT is contained in the IDT register (IDTR). 


Gate descriptors in the IDT can be interrupt, trap, or task gate descriptors. To access an interrupt or exception 
handler, the processor first receives an interrupt vector from internal hardware, an external interrupt controller, or 
from software by means of an INT n, INTO, INT3, INT1, or BOUND instruction. The interrupt vector provides an 
index into the IDT. If the selected gate descriptor is an interrupt gate or a trap gate, the associated handler proce- 
dure is accessed in a manner similar to calling a procedure through a call gate. If the descriptor is a task gate, the 
handler is accessed through a task switch. 


2.1.4.1 Interrupt and Exception Handling IA-32e Mode 


In IA-32e mode, interrupt gate descriptors are expanded to 16 bytes to support 64-bit base addresses. This is true 
for 64-bit mode and compatibility mode. 


The IDTR register is expanded to hold a 64-bit base address. Task gates are not supported. 
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2.1.5 Memory Management 


System architecture supports either direct physical addressing of memory or virtual memory (through paging). 
When physical addressing is used, a linear address is treated as a physical address. When paging is used: all code, 
data, stack, and system segments (including the GDT and IDT) can be paged with only the most recently accessed 
pages being held in physical memory. 


The location of pages (sometimes called page frames) in physical memory is contained in the paging structures. 
These structures reside in physical memory (see Figure 2-1 for the case of 32-bit paging). 


The base physical address of the paging-structure hierarchy is contained in control register CR3. The entries in the 
paging structures determine the physical address of the base of a page frame, access rights and memory manage- 
ment information. 


To use this paging mechanism, a linear address is broken into parts. The parts provide separate offsets into the 
paging structures and the page frame. A system can have a single hierarchy of paging structures or several. For 
example, each task can have its own hierarchy. 


2.1.5.1 Memory Management in IA-32e Mode 


In IA-32e mode, physical memory pages are managed by a set of system data structures. In both compatibility 
mode and 64-bit mode, four or five levels of system data structures are used (see Chapter 4, “Paging”). These 
include the following: 


° The page map level 5 (PMLS5) — An entry in the PML5 table contains the physical address of the base of a 
PML4 table, access rights, and memory management information. The base physical address of the PML5 table 
is stored in CR3. The PMLS table is used only with 5-level paging. 


¢ A page map level 4 (PML4) — An entry in a PML4 table contains the physical address of the base of a page 
directory pointer table, access rights, and memory management information. With 4-level paging, there is only 
one PML4 table and its base physical address is stored in CR3. 


° <A set of page directory pointer tables — An entry in a page directory pointer table contains the physical 
address of the base of a page directory table, access rights, and memory management information. 

° Sets of page directories — An entry in a page directory table contains the physical address of the base of a 
page table, access rights, and memory management information. 


° Sets of page tables — An entry in a page table contains the physical address of a page frame, access rights, 
and memory management information. 


2.1.6 System Registers 

To assist in initializing the processor and controlling system operations, the system architecture provides system 

flags in the EFLAGS register and several system registers: 

¢ The system flags and IOPL field in the EFLAGS register control task and mode switching, interrupt handling, 
instruction tracing, and access rights. See also: Section 2.3, “System Flags and Fields in the EFLAGS Register.” 


¢ The control registers (CRO, CR2, CR3, and CR4) contain a variety of flags and data fields for controlling system- 
level operations. Other flags in these registers are used to indicate support for specific processor capabilities 
within the operating system or executive. See also: Section 2.5, “Control Registers” and Section 2.6, “Extended 
Control Registers (Including XCRO).” 


¢ The debug registers (not shown in Figure 2-1) allow the setting of breakpoints for use in debugging programs 
and systems software. See also: Chapter 17, “Debug, Branch Profile, TSC, and Intel® Resource Director 
Technology (Intel® RDT) Features.” 


¢ The GDTR, LDTR, and IDTR registers contain the linear addresses and sizes (limits) of their respective tables. 
See also: Section 2.4, ‘Memory-Management Registers.” 


¢ The task register contains the linear address and size of the TSS for the current task. See also: Section 2.4, 
“Memory-Management Registers.” 


° Model-specific registers (not shown in Figure 2-1). 
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The model-specific registers (MSRs) are a group of registers available primarily to operating-system or executive 
procedures (that is, code running at privilege level 0). These registers control items such as the debug extensions, 
the performance-monitoring counters, the machine- check architecture, and the memory type ranges (MTRRs). 


The number and function of these registers varies among different members of the Intel 64 and IA-32 processor 
families. See also: Section 9.4, “Model-Specific Registers (MSRs),” and Chapter 2, “Model-Specific Registers 
(MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 


Most systems restrict access to system registers (other than the EFLAGS register) by application programs. 
Systems can be designed, however, where all programs and procedures run at the most privileged level (privilege 
level 0). In such a case, application programs would be allowed to modify the system registers. 


2.1.6.1 System Registers in |A-32e Mode 


In IA-32e mode, the four system-descriptor-table registers (GDTR, IDTR, LDTR, and TR) are expanded in hardware 
to hold 64-bit base addresses. EFLAGS becomes the 64-bit RFLAGS register. CRO-CR4 are expanded to 64 bits. 
CR8 becomes available. CR8 provides read-write access to the task priority register (TPR) so that the operating 
system can control the priority classes of external interrupts. 


In 64-bit mode, debug registers DRO-DR7 are 64 bits. In compatibility mode, address-matching in DRO-DR3 is 
also done at 64-bit granularity. 


On systems that support IA-32e mode, the extended feature enable register (IA32_EFER) is available. This model- 
specific register controls activation of IA-32e mode and other IA-32e mode operations. In addition, there are 
several model-specific registers that govern IA-32e mode instructions: 


° IA32_KERNEL_GS_BASE — Used by SWAPGS instruction. 
° IA32_LSTAR — Used by SYSCALL instruction. 

° IA32_FMASK — Used by SYSCALL instruction. 

¢ IA32_STAR — Used by SYSCALL and SYSRET instruction. 


2.1.7 Other System Resources 


Besides the system registers and data structures described in the previous sections, system architecture provides 
the following additional resources: 


* Operating system instructions (see also: Section 2.8, “System Instruction Summary”). 
¢ Performance-monitoring counters (not shown in Figure 2-1). 
¢ Internal caches and buffers (not shown in Figure 2-1). 


Performance-monitoring counters are event counters that can be programmed to count processor events such as 
the number of instructions decoded, the number of interrupts received, or the number of cache loads. 


The processor provides several internal caches and buffers. The caches are used to store both data and instruc- 
tions. The buffers are used to store things like decoded addresses to system and application segments and write 
operations waiting to be performed. See also: Chapter 11, “Memory Cache Control.” 


2.2 MODES OF OPERATION 


The IA-32 architecture supports three operating modes and one quasi-operating mode: 


° Protected mode — This is the native operating mode of the processor. It provides a rich set of architectural 
features, flexibility, high performance and backward compatibility to existing software base. 


¢ Real-address mode — This operating mode provides the programming environment of the Intel 8086 
processor, with a few extensions (such as the ability to switch to protected or system management mode). 


¢ System management mode (SMM) — SMM is a standard architectural feature in all IA-32 processors, 
beginning with the Intel386 SL processor. This mode provides an operating system or executive with a 
transparent mechanism for implementing power management and OEM differentiation features. SMM is 
entered through activation of an external system interrupt pin (SMI#), which generates a system management 
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interrupt (SMI). In SMM, the processor switches to a separate address space while saving the context of the 
currently running program or task. SMM-specific code may then be executed transparently. Upon returning 
from SMM, the processor is placed back into its state prior to the SMI. 


° Virtual-8086 mode — In protected mode, the processor supports a quasi-operating mode known as virtual- 
8086 mode. This mode allows the processor execute 8086 software in a protected, multitasking environment. 


Intel 64 architecture supports all operating modes of IA-32 architecture and IA-32e modes: 


® IA-32e mode — In IA-32e mode, the processor supports two sub-modes: compatibility mode and 64-bit 
mode. 64-bit mode provides 64-bit linear addressing and support for physical address space larger than 64 
GBytes. Compatibility mode allows most legacy protected-mode applications to run unchanged. 


Figure 2-3 shows how the processor moves between operating modes. 


SMI# 
Real-Address 
Mode 
Reset 
or 
Reset or ‘= RSM 
=) PES 
SMI# 
Reset > 
is RSM System 
Management 
LME=1, CRO.PG=1* — onary Mode 


A 
See** 
=1 


_ 
~Rsm 
VM=0 VM= 
* See Section 9.8.5 
irtual-8086 See Section 9.8.5.4 


Mode 


Figure 2-3. Transitions Among the Processor's Operating Modes 


The processor is placed in real-address mode following power-up or a reset. The PE flag in control register CRO then 
controls whether the processor is operating in real-address or protected mode. See also: Section 9.9, “Mode 
Switching.” and Section 4.1.2, “Paging-Mode Enabling.” 


The VM flag in the EFLAGS register determines whether the processor is operating in protected mode or virtual- 
8086 mode. Transitions between protected mode and virtual-8086 mode are generally carried out as part of a task 
switch or a return from an interrupt or exception handler. See also: Section 19.2.5, “Entering Virtual-8086 Mode.” 


The LMA bit (IA32_EFER.LMA[bit 10]) determines whether the processor is operating in IA-32e mode. When 
running in IA-32e mode, 64-bit or compatibility sub-mode operation is determined by CS.L bit of the code segment. 
The processor enters into IA-32e mode from protected mode by enabling paging and setting the LME bit 
(IA32_EFER.LME[bit 8]). See also: Chapter 9, “Processor Management and Initialization.” 


The processor switches to SMM whenever it receives an SMI while the processor is in real-address, protected, 
virtual-8086, or IA-32e modes. Upon execution of the RSM instruction, the processor always returns to the mode 
it was in when the SMI occurred. 
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Extended Feature Enable Register 


The IA32_EFER MSR provides several fields related to IA-32e mode enabling and operation. It also provides one 
field that relates to page-access right modification (see Section 4.6, “Access Rights”). The layout of the 
IA32_EFER MSR is shown in Figure 2-4. 


63 1211109 8 7 1 0 


IA32_EFER 


Execute Disable Bit Enable 


IA-32e Mode Active 


IA-32e Mode Enable 


SYSCALL Enable 


Reserved 
Figure 2-4. IA32_EFER MSR Layout 
Table 2-1. IA32_EFER MSR Information 

Bit Description 
0 SYSCALL Enable: IA32_EFER.SCE (R/W) 

Enables SYSCALL/SYSRET instructions in 64-bit mode. 
7 Reserved. 
8 IA-32e Mode Enable: IA32_EFER.LME (R/W) 

Enables I|A-32e mode operation. 
9 Reserved. 
10 IA-32e Mode Active: IA32_EFER.LMA (R) 

Indicates |A-32e mode is active when set. 
11 Execute Disable Bit Enable: IA32_EFER.NXE (R/W) 

Enables page access restriction by preventing instruction fetches from PAE pages with the XD bit set (See Section 4.6). 
63:12 Reserved. 

2.3 SYSTEM FLAGS AND FIELDS IN THE EFLAGS REGISTER 


The system flags and IOPL field of the EFLAGS register control I/O, maskable hardware interrupts, debugging, task 
switching, and the virtual-8086 mode (see Figure 2-5). Only privileged code (typically operating system or execu- 
tive code) should be allowed to modify these bits. 


The system flags and IOPL are: 


TF 


Trap (bit 8) — Set to enable single-step mode for debugging; clear to disable single-step mode. In single- 
step mode, the processor generates a debug exception after each instruction. This allows the execution 
state of a program to be inspected after each instruction. If an application program sets the TF flag using a 
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IF 


IOPL 


NT 


RF 


VM 


POPF, POPFD, or IRET instruction, a debug exception is generated after the instruction that follows the 
POPF, POPFD, or IRET. 


31 22 212019 18 17 16 15 14131211109 876543210 
Reserved (set to 0) VVfalviglols| © folo)s|r}s|ztolalole) 4c 
D ciMlE\°|T] p JElElFlFlF le) OlF| le] te 


ID — Identification Flag 

VIP — Virtual Interrupt Pending 
VIF — Virtual Interrupt Flag 
AC — Alignment Check / Access Control —— 
VM — Virtual-8086 Mode 
RF — Resume Flag 
NT — Nested Task Flag 
IOPL— I/O Privilege Level 
IF — Interrupt Enable Flag 
TF — Trap Flag 


Reserved 


Figure 2-5. System Flags in the EFLAGS Register 


Interrupt enable (bit 9) — Controls the response of the processor to maskable hardware interrupt 
requests (see also: Section 6.3.2, “Maskable Hardware Interrupts”). The flag is set to respond to maskable 
hardware interrupts; cleared to inhibit maskable hardware interrupts. The IF flag does not affect the gener- 
ation of exceptions or nonmaskable interrupts (NMI interrupts). The CPL, IOPL, and the state of the VME 
flag in control register CR4 determine whether the IF flag can be modified by the CLI, STI, POPF, POPFD, 
and IRET. 


I/O privilege level field (bits 12 and 13) — Indicates the I/O privilege level (IOPL) of the currently 
running program or task. The CPL of the currently running program or task must be less than or equal to 
the IOPL to access the I/O address space. The POPF and IRET instructions can modify this field only when 
operating at a CPL of 0. 


The IOPL is also one of the mechanisms that controls the modification of the IF flag and the handling of 
interrupts in virtual-8086 mode when virtual mode extensions are in effect (when CR4.VME = 1). See also: 
Chapter 19, “Input/Output,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 1. 


Nested task (bit 14) — Controls the chaining of interrupted and called tasks. The processor sets this flag 
on calls to a task initiated with a CALL instruction, an interrupt, or an exception. It examines and modifies 
this flag on returns from a task initiated with the IRET instruction. The flag can be explicitly set or cleared 
with the POPF/POPFD instructions; however, changing to the state of this flag can generate unexpected 
exceptions in application programs. 


See also: Section 7.4, “Task Linking.” 


Resume (bit 16) — Controls the processor’s response to instruction-breakpoint conditions. When set, this 
flag temporarily disables debug exceptions (#DB) from being generated for instruction breakpoints 
(although other exception conditions can cause an exception to be generated). When clear, instruction 
breakpoints will generate debug exceptions. 


The primary function of the RF flag is to allow the restarting of an instruction following a debug exception 
that was caused by an instruction breakpoint condition. Here, debug software must set this flag in the 
EFLAGS image on the stack just prior to returning to the interrupted program with IRETD (to prevent the 
instruction breakpoint from causing another debug exception). The processor then automatically clears 
this flag after the instruction returned to has been successfully executed, enabling instruction breakpoint 
faults again. 


See also: Section 17.3.1.1, “Instruction-Breakpoint Exception Condition.” 
Virtual-8086 mode (bit 17) — Set to enable virtual-8086 mode; clear to return to protected mode. 
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See also: Section 19.2.1, “Enabling Virtual-8086 Mode.” 


AC Alignment check or access control (bit 18) — If the AM bit is set in the CRO register, alignment 
checking of user-mode data accesses is enabled if and only if this flag is 1. An alignment-check exception 
is generated when reference is made to an unaligned operand, such as a word at an odd byte address ora 
doubleword at an address which is not an integral multiple of four. Alignment-check exceptions are gener- 
ated only in user mode (privilege level 3). Memory references that default to privilege level 0, such as 
segment descriptor loads, do not generate this exception even when caused by instructions executed in 
user-mode. 


The alignment-check exception can be used to check alignment of data. This is useful when exchanging 
data with processors which require all data to be aligned. The alignment-check exception can also be used 
by interpreters to flag some pointers as special by misaligning the pointer. This eliminates overhead of 
checking each pointer and only handles the special pointer when used. 


If the SMAP bit is set in the CR4 register, explicit supervisor-mode data accesses to user-mode pages are 
allowed if and only if this bit is 1. See Section 4.6, “Access Rights.” 


VIF Virtual Interrupt (bit 19) — Contains a virtual image of the IF flag. This flag is used in conjunction with 
the VIP flag. The processor only recognizes the VIF flag when either the VME flag or the PVI flag in control 
register CR4 is set and the IOPL is less than 3. (The VME flag enables the virtual-8086 mode extensions; 
the PVI flag enables the protected-mode virtual interrupts.) 


See also: Section 19.3.3.5, “Method 6: Software Interrupt Handling,” and Section 19.4, “Protected-Mode 
Virtual Interrupts.” 


VIP Virtual interrupt pending (bit 20) — Set by software to indicate that an interrupt is pending; cleared to 
indicate that no interrupt is pending. This flag is used in conjunction with the VIF flag. The processor reads 
this flag but never modifies it. The processor only recognizes the VIP flag when either the VME flag or the 
PVI flag in control register CR4 is set and the IOPL is less than 3. The VME flag enables the virtual-8086 
mode extensions; the PVI flag enables the protected-mode virtual interrupts. 


See Section 19.3.3.5, “Method 6: Software Interrupt Handling,” and Section 19.4, “Protected-Mode Virtual 
Interrupts.” 


ID Identification (bit 21) — The ability of a program or procedure to set or clear this flag indicates support 
for the CPUID instruction. 


2.3.1 System Flags and Fields in I|A-32e Mode 


In 64-bit mode, the RFLAGS register expands to 64 bits with the upper 32 bits reserved. System flags in RFLAGS 
(64-bit mode) or EFLAGS (compatibility mode) are shown in Figure 2-5. 


In IA-32e mode, the processor does not allow the VM bit to be set because virtual-8086 mode is not supported 
(attempts to set the bit are ignored). Also, the processor will not set the NT bit. The processor does, however, allow 
software to set the NT bit (note that an IRET causes a general protection fault in IA-32e mode if the NT bit is set). 


In IA-32e mode, the SYSCALL/SYSRET instructions have a programmable method of specifying which bits are 
cleared in RFLAGS/EFLAGS. These instructions save/restore EFLAGS/RFLAGS. 


2.4 MEMORY-MANAGEMENT REGISTERS 


The processor provides four memory-management registers (GDTR, LDTR, IDTR, and TR) that specify the locations 
of the data structures which control segmented memory management (see Figure 2-6). Special instructions are 
provided for loading and storing these registers. 
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System Table Registers 


47(79) 1615 0 
GDTR 32(64)-bit Linear Base Address 16-Bit Table Limit 
IDTR 32(64)-bit Linear Base Address 16-Bit Table Limit 
System Segment Segment Descriptor Registers (Automatically Loaded) 
Registers 
15 0 Attributes 
eae Seg. Sel. 32(64)-bit Linear Base Address Segment Limit 
LDTR Seg. Sel. 32(64)-bit Linear Base Address Segment Limit 


Figure 2-6. Memory Management Registers 


2.4.1 Global Descriptor Table Register (GDTR) 


The GDTR register holds the base address (32 bits in protected mode; 64 bits in IA-32e mode) and the 16-bit table 
limit for the GDT. The base address specifies the linear address of byte 0 of the GDT; the table limit specifies the 
number of bytes in the table. 


The LGDT and SGDT instructions load and store the GDTR register, respectively. On power up or reset of the 
processor, the base address is set to the default value of 0 and the limit is set to OFFFFH. A new base address must 
be loaded into the GDTR as part of the processor initialization process for protected-mode operation. 


See also: Section 3.5.1, “Segment Descriptor Tables.” 


2.4.2 Local Descriptor Table Register (LDTR) 


The LDTR register holds the 16-bit segment selector, base address (32 bits in protected mode; 64 bits in IA-32e 
mode), segment limit, and descriptor attributes for the LDT. The base address specifies the linear address of byte 
0 of the LDT segment; the segment limit specifies the number of bytes in the segment. See also: Section 3.5.1, 
“Segment Descriptor Tables.” 


The LLDT and SLDT instructions load and store the segment selector part of the LDTR register, respectively. The 
segment that contains the LDT must have a segment descriptor in the GDT. When the LLDT instruction loads a 
segment selector in the LDTR: the base address, limit, and descriptor attributes from the LDT descriptor are auto- 
matically loaded in the LDTR. 


When a task switch occurs, the LDTR is automatically loaded with the segment selector and descriptor for the LDT 
for the new task. The contents of the LDTR are not automatically saved prior to writing the new LDT information 
into the register. 


On power up or reset of the processor, the segment selector and base address are set to the default value of 0 and 
the limit is set to OFFFFH. 


2.4.3 IDTR Interrupt Descriptor Table Register 


The IDTR register holds the base address (32 bits in protected mode; 64 bits in IA-32e mode) and 16-bit table limit 
for the IDT. The base address specifies the linear address of byte O of the IDT; the table limit specifies the number 
of bytes in the table. The LIDT and SIDT instructions load and store the IDTR register, respectively. On power up or 
reset of the processor, the base address is set to the default value of 0 and the limit is set to OFFFFH. The base 
address and limit in the register can then be changed as part of the processor initialization process. 


See also: Section 6.10, “Interrupt Descriptor Table (IDT).” 
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2.4.4 Task Register (TR) 


The task register holds the 16-bit segment selector, base address (32 bits in protected mode; 64 bits in IA-32e 
mode), segment limit, and descriptor attributes for the TSS of the current task. The selector references the TSS 
descriptor in the GDT. The base address specifies the linear address of byte 0 of the TSS; the segment limit speci- 
fies the number of bytes in the TSS. See also: Section 7.2.4, “Task Register.” 


The LTR and STR instructions load and store the segment selector part of the task register, respectively. When the 
LTR instruction loads a segment selector in the task register, the base address, limit, and descriptor attributes from 
the TSS descriptor are automatically loaded into the task register. On power up or reset of the processor, the base 
address is set to the default value of 0 and the limit is set to OFFFFH. 


When a task switch occurs, the task register is automatically loaded with the segment selector and descriptor for 
the TSS for the new task. The contents of the task register are not automatically saved prior to writing the new TSS 
information into the register. 


2.5 CONTROL REGISTERS 


Control registers (CRO, CR1i, CR2, CR3, and CR4; see Figure 2-7) determine operating mode of the processor and 
the characteristics of the currently executing task. These registers are 32 bits in all 32-bit modes and compatibility 
mode. 


In 64-bit mode, control registers are expanded to 64 bits. The MOV CRn instructions are used to manipulate the 

register bits. Operand-size prefixes for these instructions are ignored. The following is also true: 

¢ The control registers can be read and loaded (or modified) using the move-to-or-from-control-registers forms 
of the MOV instruction. In protected mode, the MOV instructions allow the control registers to be read or loaded 
(at privilege level 0 only). This restriction means that application programs or operating-system procedures 
(running at privilege levels 1, 2, or 3) are prevented from reading or loading the control registers. 

¢ Some of the bits in CRO and CR4 are reserved and must be written with zeros. Attempting to set any reserved 
bits in CRO[31:0] is ignored. Attempting to set any reserved bits in CRO[63:32] results in a general-protection 
exception, #GP(0). Attempting to set any reserved bits in CR4 results in a general-protection exception, 
#GP(0). 

¢ All 64 bits of CR2 are writable by software. 

° Reserved bits in CR3[63:MAXPHYADDR] must be zero. Attempting to set any of them results in #GP(0). 

° The MOV CR2 instruction does not check that address written to CR2 is canonical. 


¢ A 64-bit capable processor will retain the upper 32 bits of each control register when transitioning out of IA-32e 
mode. 


* Ona 64-bit capable processor, an execution of MOV to CR outside of 64-bit mode zeros the upper 32 bits of the 
control register. 


° Register CR8 is available in 64-bit mode only. 


The control registers are summarized below, and each architecturally defined control field in these control registers 

is described individually. In Figure 2-7, the width of the register in 64-bit mode is indicated in parenthesis (except 

for CRO). 

¢ CRO — Contains system control flags that control operating mode and states of the processor. 

¢ CR1i — Reserved. 

¢ CR2 — Contains the page-fault linear address (the linear address that caused a page fault). 

¢ CR3 — Contains the physical address of the base of the paging-structure hierarchy and two flags (PCD and 
PWT). Only the most-significant bits (less the lower 12 bits) of the base address are specified; the lower 12 bits 
of the address are assumed to be 0. The first paging structure must thus be aligned to a page (4-KByte) 


boundary. The PCD and PWT flags control caching of that paging structure in the processor’s internal data 
caches (they do not control TLB caching of page-directory information). 


When using the physical address extension, the CR3 register contains the base address of the page-directory- 
pointer table. With 4-level paging and 5-level paging, the CR3 register contains the base address of the PML4 
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table and PMLS table, respectively. If PCIDs are enabled, CR3 has a format different from that illustrated in 
Figure 2-7. See Section 4.5, “4-Level Paging and 5-Level Paging.” 


See also: Chapter 4, “Paging.” 


¢ CR4 — Contains a group of flags that enable several architectural extensions, and indicate operating system or 
executive support for specific processor capabilities. Bits CR4[63:32] can only be used for IA-32e mode only 
features that are enabled after entering 64-bit mode. Bits CR4[63:32] do not have any effect outside of IA-32e 
mode. 


* CR8 — Provides read and write access to the Task Priority Register (TPR). It specifies the priority threshold 
value that operating systems use to control the priority class of external interrupts allowed to interrupt the 
processor. This register is available only in 64-bit mode. However, interrupt filtering continues to apply in 
compatibility mode. 


31 (63) 24 23 22 21 20 19 18 17 16 15 14 #13 #12 «#11 «10 9 8 x 6 5 4 3 2 4 0 
p|cj/p/s/s/, mil a ry pl)e|m/p/p],|tlely 
Reserved K|E]K C/|G/C}]A/S s|V|]M]| CR4 
Te: | ae) oss) ales ! ElefelelelFloji je 
P| P E/E P 
osxsave—_! ‘— FSGSBASE moet 
PCIDE ‘— OSXMMEXCPT 
31 (63) 12 11 5 4 3 2 0 
P| P 
Page-Directory Base c|w CR3 
D/| T 
31 (63) 0 
Page-Fault Linear Address CR2 
31 (63) 0 
CRI 
31 30 29 28 19 18 17 16 15 6 5 4 3 2 1 0 
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Reserved 


Figure 2-7. Control Registers 


The flags in control registers are: 
CRO.PG 


Paging (bit 31 of CRO) — Enables paging when set; disables paging when clear. When paging is 
disabled, all linear addresses are treated as physical addresses. The PG flag has no effect if the PE flag (bit 
0 of register CRO) is not also set; setting the PG flag when the PE flag is clear causes a general-protection 
exception (#GP). See also: Chapter 4, “Paging.” 


On Intel 64 processors, enabling and disabling IA-32e mode operation also requires modifying CRO.PG. 
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CRO.CD 


Cache Disable (bit 30 of CRO) — When the CD and NW flags are clear, caching of memory locations for 
the whole of physical memory in the processor’s internal (and external) caches is enabled. When the CD 
flag is set, caching is restricted as described in Table 11-5. To prevent the processor from accessing and 
updating its caches, the CD flag must be set and the caches must be invalidated so that no cache hits can 
occur. 


See also: Section 11.5.3, “Preventing Caching,” and Section 11.5, “Cache Control.” 


CRO.NW 
Not Write-through (bit 29 of CRO) — When the NW and CD flags are clear, write-back (for Pentium 4, 
Intel Xeon, P6 family, and Pentium processors) or write-through (for Intel486 processors) is enabled for 
writes that hit the cache and invalidation cycles are enabled. See Table 11-5 for detailed information about 
the effect of the NW flag on caching for other settings of the CD and NW flags. 


CRO.AM 
Alignment Mask (bit 18 of CRO) — Enables automatic alignment checking when set; disables alignment 
checking when clear. Alignment checking is performed only when the AM flag is set, the AC flag in the 
EFLAGS register is set, CPL is 3, and the processor is operating in either protected or virtual-8086 mode. 


CRO.WP 
Write Protect (bit 16 of CRO) — When set, inhibits supervisor-level procedures from writing into read- 
only pages; when clear, allows supervisor-level procedures to write into read-only pages (regardless of the 
U/S bit setting; see Section 4.1.3 and Section 4.6). This flag facilitates implementation of the copy-on- 
write method of creating a new process (forking) used by operating systems such as UNIX. This flag must 
be set before software can set CR4.CET, and it cannot be cleared as long as CR4.CET = 1 (see below). 


CRO.NE 
Numeric Error (bit 5 of CRO) — Enables the native (internal) mechanism for reporting x87 FPU errors 
when set; enables the PC-style x87 FPU error reporting mechanism when clear. When the NE flag is clear 
and the IGNNE# input is asserted, x87 FPU errors are ignored. When the NE flag is clear and the IGNNE# 
input is deasserted, an unmasked x87 FPU error causes the processor to assert the FERR# pin to generate 
an external interrupt and to stop instruction execution immediately before executing the next waiting 
floating-point instruction or WAIT/FWAIT instruction. 


The FERR# pin is intended to drive an input to an external interrupt controller (the FERR# pin emulates the 
ERROR# pin of the Intel 287 and Intel 387 DX math coprocessors). The NE flag, IGNNE# pin, and FERR# 
pin are used with external logic to implement PC-style error reporting. Using FERR# and IGNNE# to handle 
floating-point exceptions is deprecated by modern operating systems; this non-native approach also limits 
newer processors to operate with one logical processor active. 


See also: Section 8.7, “Handling x87 FPU Exceptions in Software” in Chapter 8, “Programming with the x87 
FPU,” and Appendix A, “EFLAGS Cross-Reference,” in the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1. 


CRO.ET 
Extension Type (bit 4 of CRO) — Reserved in the Pentium 4, Intel Xeon, P6 family, and Pentium proces- 
sors. In the Pentium 4, Intel Xeon, and P6 family processors, this flag is hardcoded to 1. In the Intel386 
and Intel486 processors, this flag indicates support of Intel 387 DX math coprocessor instructions when 
set. 


CRO.TS 
Task Switched (bit 3 of CRO) — Allows the saving of the x87 FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 
context on a task switch to be delayed until an x87 FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instruction is 
actually executed by the new task. The processor sets this flag on every task switch and tests it when 
executing x87 FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instructions. 


¢ Ifthe TS flag is set and the EM flag (bit 2 of CRO) is clear, a device-not-available exception (#NM) is 
raised prior to the execution of any x87 FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instruction; with the 
exception of PAUSE, PREFETCHhA, SFENCE, LFENCE, MFENCE, MOVNTI, CLFLUSH, CRC32, and POPCNT. 
See the paragraph below for the special case of the WAIT/FWAIT instructions. 


¢ Ifthe TS flag is set and the MP flag (bit 1 of CRO) and EM flag are clear, an #NM exception is not raised 
prior to the execution of an x87 FPU WAIT/FWAIT instruction. 
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¢ If the EM flag is set, the setting of the TS flag has no effect on the execution of x87 
FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instructions. 


Table 2-2 shows the actions taken when the processor encounters an x87 FPU instruction based on the 
settings of the TS, EM, and MP flags. Table 12-1 and 13-1 show the actions taken when the processor 
encounters an MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instruction. 


The processor does not automatically save the context of the x87 FPU, XMM, and MXCSR registers on a 
task switch. Instead, it sets the TS flag, which causes the processor to raise an #NM exception whenever it 
encounters an x87 FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instruction in the instruction stream for the 
new task (with the exception of the instructions listed above). 


The fault handler for the #NM exception can then be used to clear the TS flag (with the CLTS instruction) 
and save the context of the x87 FPU, XMM, and MXCSR registers. If the task never encounters an x87 
FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instruction, the x87 FPU/MMX/SSE/SSE2/SSE3/SSSE3/SSE4 
context is never saved. 


Table 2-2. Action Taken By x87 FPU Instructions for Different Combinations of EM, MP, and TS 


CRO Flags x87 FPU Instruction Type 

EM MP TS Floating-Point WAIT/FWAIT 
0 0 0 Execute Execute. 

0) 0 1 #NM Exception Execute. 

0 1 0 Execute Execute. 

0 1 1 #NM Exception #NM exception. 
1 0 0 #NM Exception Execute. 

1 0 1 #NM Exception Execute. 

1 1 0 #NM Exception Execute. 

1 1 1 #NM Exception #NM exception. 

CRO.EM 


Emulation (bit 2 of CRO) — Indicates that the processor does not have an internal or external x87 FPU when set; 
indicates an x87 FPU is present when clear. This flag also affects the execution of 
MMX/SSE/SSE2/SSE3/SSSE3/SSE4 instructions. 


When the EM flag is set, execution of an x87 FPU instruction generates a device-not-available exception 
(#NM). This flag must be set when the processor does not have an internal x87 FPU or is not connected to 
an external math coprocessor. Setting this flag forces all floating-point instructions to be handled by soft- 
ware emulation. Table 9-3 shows the recommended setting of this flag, depending on the IA-32 processor 
and x87 FPU or math coprocessor present in the system. Table 2-2 shows the interaction of the EM, MP, and 
TS flags. 


Also, when the EM flag is set, execution of an MMX instruction causes an invalid-opcode exception (#UD) 
to be generated (see Table 12-1). Thus, if an IA-32 or Intel 64 processor incorporates MMX technology, the 
EM flag must be set to 0 to enable execution of MMX instructions. 


Similarly for SSE/SSE2/SSE3/SSSE3/SSE4 extensions, when the EM flag is set, execution of most 
SSE/SSE2/SSE3/SSSE3/SSE4 instructions causes an invalid opcode exception (#UD) to be generated (see 
Table 13-1). If an IA-32 or Intel 64 processor incorporates the SSE/SSE2/SSE3/SSSE3/SSE4 extensions, 
the EM flag must be set to 0 to enable execution of these extensions. SSE/SSE2/SSE3/SSSE3/SSE4 
instructions not affected by the EM flag include: PAUSE, PREFETCHh, SFENCE, LFENCE, MFENCE, MOVNTI, 
CLFLUSH, CRC32, and POPCNT. 


CRO.MP 


Monitor Coprocessor (bit 1 of CRO) — Controls the interaction of the WAIT (or FWAIT) instruction with 
the TS flag (bit 3 of CRO). If the MP flag is set, a WAIT instruction generates a device-not-available exception 
(#NM) if the TS flag is also set. If the MP flag is clear, the WAIT instruction ignores the setting of the TS flag. 
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Table 9-3 shows the recommended setting of this flag, depending on the IA-32 processor and x87 FPU or 
math coprocessor present in the system. Table 2-2 shows the interaction of the MP, EM, and TS flags. 


CRO.PE 
Protection Enable (bit 0 of CRO) — Enables protected mode when set; enables real-address mode when 
clear. This flag does not enable paging directly. It only enables segment-level protection. To enable paging, 
both the PE and PG flags must be set. 


See also: Section 9.9, “Mode Switching.” 


CR3.PCD 
Page-level Cache Disable (bit 4 of CR3) — Controls the memory type used to access the first paging 
structure of the current paging-structure hierarchy. See Section 4.9, “Paging and Memory Typing”. This bit 
is not used if paging is disabled, with PAE paging, or with 4-level paging? or 5-level paging if CR4.PCIDE=1. 


CR3.PWT 
Page-level Write-Through (bit 3 of CR3) — Controls the memory type used to access the first paging 
structure of the current paging-structure hierarchy. See Section 4.9, “Paging and Memory Typing”. This bit 
is not used if paging is disabled, with PAE paging, or with 4-level paging or 5-level paging if CR4.PCIDE=1. 


CR4.VME 
Virtual-8086 Mode Extensions (bit 0 of CR4) — Enables interrupt- and exception-handling extensions 
in virtual-8086 mode when set; disables the extensions when clear. Use of the virtual mode extensions can 
improve the performance of virtual-8086 applications by eliminating the overhead of calling the virtual- 
8086 monitor to handle interrupts and exceptions that occur while executing an 8086 program and, 
instead, redirecting the interrupts and exceptions back to the 8086 program’s handlers. It also provides 
hardware support for a virtual interrupt flag (VIF) to improve reliability of running 8086 programs in multi- 
tasking and multiple-processor environments. 


See also: Section 19.3, “Interrupt and Exception Handling in Virtual-8086 Mode.” 


CR4.PVI 
Protected-Mode Virtual Interrupts (bit 1 of CR4) — Enables hardware support for a virtual interrupt 
flag (VIF) in protected mode when set; disables the VIF flag in protected mode when clear. 


See also: Section 19.4, “Protected-Mode Virtual Interrupts.” 


CR4.TSD 
Time Stamp Disable (bit 2 of CR4) — Restricts the execution of the RDTSC instruction to procedures 
running at privilege level O when set; allows RDTSC instruction to be executed at any privilege level when 
clear. This bit also applies to the RDTSCP instruction if supported (if CPUID.80000001H:EDX[27] = 1). 


CR4.DE 
Debugging Extensions (bit 3 of CR4) — References to debug registers DR4 and DR5 cause an unde- 
fined opcode (#UD) exception to be generated when set; when clear, processor aliases references to regis- 
ters DR4 and DRS for compatibility with software written to run on earlier IA-32 processors. 


See also: Section 17.2.2, “Debug Registers DR4 and DR5.” 


CR4.PSE 
Page Size Extensions (bit 4 of CR4) — Enables 4-MByte pages with 32-bit paging when set; restricts 
32-bit paging to pages of 4 KBytes when clear. 


See also: Section 4.3, “32-Bit Paging.” 


CR4.PAE 
Physical Address Extension (bit 5 of CR4) — When set, enables paging to produce physical addresses 
with more than 32 bits. When clear, restricts physical addresses to 32 bits. PAE must be set before entering 
IA-32e mode. 


See also: Chapter 4, “Paging.” 


CR4.MCE 
Machine-Check Enable (bit 6 of CR4) — Enables the machine-check exception when set; disables the 
machine-check exception when clear. 


2. Earlier versions of this manual used the term “IA-32e paging” to identify 4-level paging. 
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See also: Chapter 15, “Machine-Check Architecture.” 


CR4.PGE 
Page Global Enable (bit 7 of CR4) — (Introduced in the P6 family processors.) Enables the global page 
feature when set; disables the global page feature when clear. The global page feature allows frequently 
used or shared pages to be marked as global to all users (done with the global flag, bit 8, in a page-direc- 
tory-pointer-table entry, a page-directory entry, or a page-table entry). Global pages are not flushed from 
the translation-lookaside buffer (TLB) on a task switch or a write to register CR3. 


When enabling the global page feature, paging must be enabled (by setting the PG flag in control register 
CRO) before the PGE flag is set. Reversing this sequence may affect program correctness, and processor 
performance will be impacted. 


See also: Section 4.10, “Caching Translation Information.” 


CR4.PCE 
Performance-Monitoring Counter Enable (bit 8 of CR4) — Enables execution of the RDPMC instruc- 
tion for programs or procedures running at any protection level when set; RDPMC instruction can be 
executed only at protection level 0 when clear. 


CR4.0SFXSR 
Operating System Support for FXSAVE and FXRSTOR instructions (bit 9 of CR4) — When set, this 
flag: (1) indicates to software that the operating system supports the use of the FXSAVE and FXRSTOR 
instructions, (2) enables the FXSAVE and FXRSTOR instructions to save and restore the contents of the 
XMM and MXCSR registers along with the contents of the x87 FPU and MMX registers, and (3) enables the 
processor to execute SSE/SSE2/SSE3/SSSE3/SSE4 instructions, with the exception of the PAUSE, 
PREFETCHh, SFENCE, LFENCE, MFENCE, MOVNTI, CLFLUSH, CRC32, and POPCNT. 


If this flag is clear, the FKSAVE and FXRSTOR instructions will save and restore the contents of the x87 FPU 
and MMX registers, but they may not save and restore the contents of the XMM and MXCSR registers. Also, 
the processor will generate an invalid opcode exception (#UD) if it attempts to execute any 
SSE/SSE2/SSE3 instruction, with the exception of PAUSE, PREFETCHh, SFENCE, LFENCE, MFENCE, 
MOVNTI, CLFLUSH, CRC32, and POPCNT. The operating system or executive must explicitly set this flag. 


NOTE 


CPUID feature flag FXSR indicates availability of the FKSAVE/FXRSTOR instructions. The OSFXSR 
bit provides operating system software with a means of enabling FKSAVE/FXRSTOR to save/restore 
the contents of the X87 FPU, XMM and MXCSR registers. Consequently OSFXSR bit indicates that 
the operating system provides context switch support for SSE/SSE2/SSE3/SSSE3/SSE4. 


CR4.0SXMMEXCPT 
Operating System Support for Unmasked SIMD Floating-Point Exceptions (bit 10 of CR4) — 
When set, indicates that the operating system supports the handling of unmasked SIMD floating-point 
exceptions through an exception handler that is invoked when a SIMD floating-point exception (#XM) is 
generated. SIMD floating-point exceptions are only generated by SSE/SSE2/SSE3/SSE4.1 SIMD floating- 
point instructions. 


The operating system or executive must explicitly set this flag. If this flag is not set, the processor will 
generate an invalid opcode exception (#UD) whenever it detects an unmasked SIMD floating-point excep- 
tion. 

CR4.UMIP 
User-Mode Instruction Prevention (bit 11 of CR4) — When set, the following instructions cannot be 
executed if CPL > 0: SGDT, SIDT, SLDT, SMSW, and STR. An attempt at such execution causes a general- 
protection exception (#GP). 

CR4.LA57 
57-bit linear addresses (bit 12 of CR4) — When set in IA-32e mode, the processor uses 5-level paging 
to translate 57-bit linear addresses. When clear in IA-32e mode, the processor uses 4-level paging to 
translate 48-bit linear addresses. This bit cannot be modified in IA-32e mode. 


See also: Chapter 4, “Paging.” 
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CR4.VMXE 
VMX-Enable Bit (bit 13 of CR4) — Enables VMX operation when set. See Chapter 22, “Introduction to 
Virtual Machine Extensions.” 


CR4.SMXE 
SMX-Enable Bit (bit 14 of CR4) — Enables SMX operation when set. See Chapter 6, “Safer Mode Exten- 
sions Reference” of Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2D. 


CR4.FSGSBASE 
FSGSBASE-Enable Bit (bit 16 of CR4) — Enables the instructions RDFSBASE, RDGSBASE, WRFSBASE, 
and WRGSBASE. 


CR4.PCIDE 
PCID-Enable Bit (bit 17 of CR4) — Enables process-context identifiers (PCIDs) when set. See Section 
4.10.1, “Process-Context Identifiers (PCIDs)”. Applies only in IA-32e mode (if IA32_EFER.LMA = 1). 


CR4.0SXSAVE 
XSAVE and Processor Extended States-Enable Bit (bit 18 of CR4) — When set, this flag: (1) indi- 
cates (via CPUID.01H:ECX.OSXSAVE[bit 27]) that the operating system supports the use of the XGETBV, 
XSAVE and XRSTOR instructions by general software; (2) enables the XSAVE and XRSTOR instructions to 
save and restore the x87 FPU state (including MMX registers), the SSE state (XMM registers and MXCSR), 
along with other processor extended states enabled in XCRO; (3) enables the processor to execute XGETBV 
and XSETBV instructions in order to read and write XCRO. See Section 2.6 and Chapter 13, “System 
Programming for Instruction Set Extensions and Processor Extended States”. 


CR4.KL 
Key-Locker-Enable Bit (bit 19 of CR4) — When set, the LOADIWKEY instruction is enabled; in addition, 
if support for the AES Key Locker instructions has been activated by system firmware, 
CPUID.19H:EBX.AESKLE[bit 0] is enumerated as 1 and the AES Key Locker instructions are enabled.? 
When clear, CPUID.19H:EBX.AESKLE[bit 0] is enumerated as 0 and execution of any Key Locker instruction 
causes an invalid-opcode exception (#UD). 


CR4.SMEP 
SMEP-Enable Bit (bit 20 of CR4) — Enables supervisor-mode execution prevention (SMEP) when set. 
See Section 4.6, “Access Rights”. 


CR4.SMAP 
SMAP-Enable Bit (bit 21 of CR4) — Enables supervisor-mode access prevention (SMAP) when set. See 
Section 4.6, “Access Rights.” 


CR4.PKE 
Enable protection keys for user-mode pages (bit 22 of CR4) — 4-level paging and 5-level paging 
associate each user-mode linear address with a protection key. When set, this flag indicates (via 
CPUID.(EAX=07H,ECX=0H):ECX.OSPKE [bit 4]) that the operating system supports use of the PKRU 
register to specify, for each protection key, whether user-mode linear addresses with that protection key 
can be read or written. This bit also enables access to the PKRU register using the RDPKRU and WRPKRU 
instructions. 


CR4.CET 
Control-flow Enforcement Technology (bit 23 of CR4) — Enables control-flow enforcement tech- 
nology when set. See Chapter 18, “Control-flow Enforcement Technology (CET)” of the IA-32 Intel® Archi- 
tecture Software Developer’s Manual, Volume 1. This flag can be set only if CRO.WP is set, and it must be 
clear before CRO.WP can be cleared (see below). 


CR4.PKS 
Enable protection keys for supervisor-mode pages (bit 24 of CR4) — 4-level paging and 5-level 
paging associate each supervisor-mode linear address with a protection key. When set, this flag allows use 
of the IA32_PKRS MSR to specify, for each protection key, whether supervisor-mode linear addresses with 
that protection key can be read or written. 


3. Software can check CPUID.1 9H:EBX.AESKLE[bit 0] after setting CR4.KL to determine whether the AES Key Locker instructions have 
been enabled. Note that some processors may allow enabling of those instructions without activation by system firmware. Some 
processors may not support use of the AES Key Locker instructions in system-management mode (SMM). Those processors enumer- 
ate CPUID.1 9H:EBX.AESKLE[bit 0] as O in SMM regardless of the setting of CR4.KL. 
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CR8.TPL 
Task Priority Level (bit 3:0 of CR8) — This sets the threshold value corresponding to the highest- 
priority interrupt to be blocked. A value of 0 means all interrupts are enabled. This field is available in 64- 
bit mode. A value of 15 means all interrupts will be disabled. 


2.5.1 CPUID Qualification of Control Register Flags 


Not all flags in control register CR4 are implemented on all processors. With the exception of the PCE flag, they can 
be qualified with the CPUID instruction to determine if they are implemented on the processor before they are 
used. 


The CR8 register is available on processors that support Intel 64 architecture. 


2.6 EXTENDED CONTROL REGISTERS (INCLUDING XCRO) 


If CPUID.O1H:ECX.XSAVE[bit 26] is 1, the processor supports one or more extended control registers (XCRs). 
Currently, the only such register defined is XCRO. This register specifies the set of processor state components for 
which the operating system provides context management, e.g. x87 FPU state, SSE state, AVX state. The OS 
programs XCRO to reflect the features for which it provides context management. 


63 9 76543 210 


Reserved (must be 0) 1 


Reserved for XCRO bit vector expansion 
Reserved / Future processor extended states 
PKRU state 
Hi16_ZMM state 
ZMM_Hi256 state 
Opmask state 
BNDCSR state 
BNDREG state 
AVX state 
SSE state 
x87 FPU/MMxX state (must be 1) 


Figure 2-8. XCRO 


Software can access XCRO only if CR4.0SXSAVE[bit 18] = 1. (This bit is also readable as 
CPUID.01H:ECX.OSXSAVE[bit 27].) Software can use CPUID leaf function ODH to enumerate the bits in XCRO that 
the processor supports (see CPUID instruction in Intel® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 2A). Each supported state component is represented by a bit in XCRO. System software enables state 
components by loading an appropriate bit mask value into XCRO using the XSETBV instruction. 


As each bit in XCRO (except bit 63) corresponds to a processor state component, XCRO thus provides support for 
up to 63 sets of processor state components. Bit 63 of XCRO is reserved for future expansion and will not represent 
a processor state component. 


Currently, XCRO defines support for the following state components: 
*  XCRO.X87 (bit 0): This bit O must be 1. An attempt to write 0 to this bit causes a #GP exception. 


¢ XCRO.SSE (bit 1): If 1, the XSAVE feature set can be used to manage MXCSR and the XMM registers (XMMO- 
XMM15 in 64-bit mode; otherwise XMMO-XMM7). 


¢ XCRO.AVX (bit 2): If 1, AVX instructions can be executed and the XSAVE feature set can be used to manage the 
upper halves of the YMM registers (YMMO-YMM15 in 64-bit mode; otherwise YMMO-YMM7). 
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¢ XCRO.BNDREG (bit 3): If 1, MPX instructions can be executed and the XSAVE feature set can be used to 
manage the bounds registers BNDO-BND3. 


¢ XCRO.BNDCSR (bit 4): If 1, MPX instructions can be executed and the XSAVE feature set can be used to 
manage the BNDCFGU and BNDSTATUS registers. 


¢ XCRO.opmask (bit 5): If 1, AVX-512 instructions can be executed and the XSAVE feature set can be used to 
manage the opmask registers kO-k7. 


¢ XCRO.ZMM_Hi256 (bit 6): If 1, AVX-512 instructions can be executed and the XSAVE feature set can be used 
to manage the upper halves of the lower ZMM registers (ZMMO-ZMM15 in 64-bit mode; otherwise ZMMO- 
ZMM7). 


¢ = XCRO.Hi16_ZMM (bit 7): If 1, AVX-512 instructions can be executed and the XSAVE feature set can be used to 
manage the upper ZMM registers (ZMM16-ZMM31, only in 64-bit mode). 


¢ XCRO.PKRU (bit 9): If 1, the XSAVE feature set can be used to manage the PKRU register (see Section 2.7). 


An attempt to use XSETBV to write to XCRO results in general-protection exceptions (#GP) if it would do any of the 
following: 


° Set a bit reserved in XCRO for a given processor (as determined by the contents of EAX and EDX after executing 
CPUID with EAX=0DH, ECX= OH). 


° Clear XCRO.x87. 

¢ Clear XCRO.SSE and set XCRO.AVX. 

¢ Clear XCRO.AVX and set any of XCRO.opmask, XCRO.ZMM_Hi256, and XCRO.Hi16_ZMM. 

° Set either XCRO.BNDREG and XCRO.BNDCSR while not setting the other. 

° Set any of XCRO.opmask, XCRO.ZMM_Hi256, and XCRO.Hi16_ZMM while not setting all of them. 
After reset, all bits (except bit 0) in XCRO are cleared to zero; XCRO[O] is set to 1. 


27 PROTECTION-KEY RIGHTS REGISTERS (PKRU AND IA32_PKRS) 


Processors may support either or both of two protection-key rights registers: PKRU for user-mode pages and the 
IA32_PKRS MSR (MSR index 6E1H) for supervisor-mode pages. 4-level paging and 5-level paging associate a 4-bit 
protection key with each page. The protection-key rights registers determine accessibility based on a page’s 
protection key. 


If CPUID.(EAX=07H,ECX=0H):ECX.PKU [bit 3] = 1, the processor supports the protection-key feature for user- 
mode pages. When CR4.PKE = 1, software can use the protection-key rights register for user pages (PKRU) 
to specify the access rights for user-mode pages for each protection key. 

If CPUID.(EAX=07H,ECX=0H):ECX.PKS [bit 31] = 1, the processor supports the protection-key feature for super- 


visor-mode pages. When CR4.PKS = 1, software can use the protection-key rights register for supervisor 
pages (the IA32_PKRS MSR) to specify the access rights for supervisor-mode pages for each protection key. 


31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 1413121110 9 8 7 6 5 4 3 2 1 0 _ Bit Position 
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Figure 2-9. Format of Protection-Key Rights Registers 
The format of each protection-key rights register is given in Figure 2-9. Each contains 16 pairs of disable controls 
to prevent data accesses to linear addresses (user-mode or supervisor-mode, depending on the register) based on 
their protection keys. Each protection key / (0 </ < 15) is associated with two bits in each protection-key rights 
register: 
° Bit 2/, shown as “ADi” (access disable): if set, the processor prevents any data accesses to linear addresses 
(user-mode or supervisor-mode, depending on the register) with protection key /. 
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° Bit 2i+1, shown as “WDi” (write disable): if set, the processor prevents write accesses to linear addresses 
(user-mode or supervisor-mode, depending on the register) with protection key i. 


(Bits 63:32 of the IA32_PKRS MSR are reserved and must be zero.) 


See Section 4.6.2, “Protection Keys,” for details of how the processor uses the protection-key rights registers to 
control accesses to linear addresses. 


Software can read and write PKRU using the RDPKRU and WRPKRU instructions. The IA32_PKRS MSR can be read 
and written with the RDMSR and WRMSR instructions. Writes to the IA32_PKRS MSR using WRMSR are not serial- 
izing. 


2.8 SYSTEM INSTRUCTION SUMMARY 


System instructions handle system-level functions such as loading system registers, managing the cache, 
managing interrupts, or setting up the debug registers. Many of these instructions can be executed only by oper- 
ating-system or executive procedures (that is, procedures running at privilege level 0). Others can be executed at 
any privilege level and are thus available to application programs. 


Table 2-3 lists the system instructions and indicates whether they are available and useful for application programs. 
These instructions are described in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volumes 
2A, 2B, 2C & 2D. 


Table 2-3. Summary of System Instructions 


Useful to Protected from 

Instruction Description Application? Application? 

LLDT Load LDT Register No Yes 
SLDT Store LDT Register No If CR4.UMIP = 1 
LGDT Load GDT Register No Yes 
SGDT Store GDT Register No If CR4.UMIP = 1 
LTR Load Task Register No Yes 

STR Store Task Register No If CR4.UMIP = 1 
LIDT Load IDT Register No Yes 

SIDT Store IDT Register No If CR4.UMIP = 1 
MOV CRn Load and store control registers No Yes 
SMSW Store MSW Yes If CR4.UMIP = 1 
LMSW Load MSW No Yes 

CLTS Clear TS flag in CRO No Yes 
ARPL Adjust RPL Yes!.5 No 

LAR Load Access Rights Yes No 

LSL Load Segment Limit Yes No 
VERR Verify for Reading Yes No 
VERW Verify for Writing Yes No 

MOV DRn Load and store debug registers No Yes 

INVD Invalidate cache, no writeback No Yes 
WBINVD Invalidate cache, with writeback No Yes 
INVLPG Invalidate TLB entry No Yes 

HLT Halt Processor No Yes 
LOCK (Prefix) Bus Lock Yes No 
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Table 2-3. Summary of System Instructions (Contd.) 


Useful to Protected from 
Instruction Description Application? Application? 
RSM Return from system management mode No Yes 
RDMSR2 Read Model-Specific Registers No Yes 
WRMSR?3 Write Model-Specific Registers No Yes 
RDPMC* Read Performance-Monitoring Counter Yes Yes? 
RDTSC3 Read Time-Stamp Counter Yes Yes? 
RDTSCP? Read Serialized Time-Stamp Counter Yes Yes? 
XGETBV Return the state of XCRO Yes No 
XSETBV Enable one or more processor extended states No® Yes 


NOTES: 

1. Useful to application programs running at a CPL of 1 or 2. 

2. The TSD and PCE flags in control register CR4 control access to these instructions by application programs running at a CPL of 3. 

3. These instructions were introduced into the IA-32 Architecture with the Pentium processor. 

4. This instruction was introduced into the IA-32 Architecture with the Pentium Pro processor and the Pentium processor with MMX technol- 
ogy. 

5. This instruction is not supported in 64-bit mode. 

6. Application uses XGETBV to query which set of processor extended states are enabled. 

7. RDTSCP is introduced in Intel Core i7 processor. 


2.8.1 Loading and Storing System Registers 

The GDTR, LDTR, IDTR, and TR registers each have a load and store instruction for loading data into and storing 
data from the register: 

¢ LGDT (Load GDTR Register) — Loads the GDT base address and limit from memory into the GDTR register. 
° SGDT (Store GDTR Register) — Stores the GDT base address and limit from the GDTR register into memory. 
¢ LIDT (Load IDTR Register) — Loads the IDT base address and limit from memory into the IDTR register. 

° SIDT (Store IDTR Register) — Stores the IDT base address and limit from the IDTR register into memory. 


¢ LLDT (Load LDTR Register) — Loads the LDT segment selector and segment descriptor from memory into 
the LDTR. (The segment selector operand can also be located in a general-purpose register. ) 


¢ SLDT (Store LDTR Register) — Stores the LDT segment selector from the LDTR register into memory or a 
general-purpose register. 


¢ LTR (Load Task Register) — Loads segment selector and segment descriptor for a TSS from memory into the 
task register. (The segment selector operand can also be located in a general-purpose register. ) 


¢ STR (Store Task Register) — Stores the segment selector for the current task TSS from the task register into 
memory or a general-purpose register. 


The LMSW (load machine status word) and SMSW (store machine status word) instructions operate on bits 0 
through 15 of control register CRO. These instructions are provided for compatibility with the 16-bit Intel 286 
processor. Programs written to run on 32-bit IA-32 processors should not use these instructions. Instead, they 
should access the control register CRO using the MOV CR instruction. 


The CLTS (clear TS flag in CRO) instruction is provided for use in handling a device-not-available exception (#NM) 
that occurs when the processor attempts to execute a floating-point instruction when the TS flag is set. This 
instruction allows the TS flag to be cleared after the x87 FPU context has been saved, preventing further #NM 
exceptions. See Section 2.5, “Control Registers,” for more information on the TS flag. 


The control registers (CRO, CR1, CR2, CR3, CR4, and CR8) are loaded using the MOV instruction. The instruction 
loads a control register from a general-purpose register or stores the content of a control register in a general- 
purpose register. 
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2.8.2 Verifying of Access Privileges 


The processor provides several instructions for examining segment selectors and segment descriptors to determine 
if access to their associated segments is allowed. These instructions duplicate some of the automatic access rights 
and type checking done by the processor, thus allowing operating-system or executive software to prevent excep- 
tions from being generated. 


The ARPL (adjust RPL) instruction adjusts the RPL (requestor privilege level) of a segment selector to match that of 
the program or procedure that supplied the segment selector. See Section 5.10.4, “Checking Caller Access 
Privileges (ARPL Instruction)” for a detailed explanation of the function and use of this instruction. Note that 
ARPL is not supported in 64-bit mode. 


The LAR (load access rights) instruction verifies the accessibility of a specified segment and loads access rights 
information from the segment’s segment descriptor into a general-purpose register. Software can then examine 
the access rights to determine if the segment type is compatible with its intended use. See Section 5.10.1, 
“Checking Access Rights (LAR Instruction)” for a detailed explanation of the function and use of this instruc- 
tion. 


The LSL (load segment limit) instruction verifies the accessibility of a specified segment and loads the segment 
limit from the segment’s segment descriptor into a general-purpose register. Software can then compare the 
segment limit with an offset into the segment to determine whether the offset lies within the segment. See Section 
5.10.3, “Checking That the Pointer Offset Is Within Limits (LSL Instruction)” for a detailed explanation of 
the function and use of this instruction. 


The VERR (verify for reading) and VERW (verify for writing) instructions verify if a selected segment is readable or 
writable, respectively, at a given CPL. See Section 5.10.2, “Checking Read/Write Rights (VERR and VERW 
Instructions)” for a detailed explanation of the function and use of these instructions. 


2.8.3 Loading and Storing Debug Registers 


Internal debugging facilities in the processor are controlled by a set of 8 debug registers (DRO-DR7). The MOV 
instruction allows setup data to be loaded to and stored from these registers. 


On processors that support Intel 64 architecture, debug registers DRO-DR7 are 64 bits. In 32-bit modes and 
compatibility mode, writes to a debug register fill the upper 32 bits with zeros. Reads return the lower 32 bits. In 
64-bit mode, the upper 32 bits of DR6-DR7 are reserved and must be written with zeros. Writing one to any of the 
upper 32 bits causes an exception, #GP(0). 


In 64-bit mode, MOV DRn instructions read or write all 64 bits of a debug register (operand-size prefixes are 
ignored). All 64 bits of DRO-DR3 are writable by software. However, MOV DRn instructions do not check that 
addresses written to DRO-DR3 are in the limits of the implementation. Address matching is supported only on valid 
addresses generated by the processor implementation. 


2.8.4 Invalidating Caches and TLBs 


The processor provides several instructions for use in explicitly invalidating its caches and TLB entries. The INVD 
(invalidate cache with no writeback) instruction invalidates all data and instruction entries in the internal caches 
and sends a signal to the external caches indicating that they should also be invalidated. 


The WBINVD (invalidate cache with writeback) instruction performs the same function as the INVD instruction, 
except that it writes back modified lines in its internal caches to memory before it invalidates the caches. After 
invalidating the caches local to the executing logical processor or processor core, WBINVD signals caches higher in 
the cache hierarchy (caches shared with the invalidating logical processor or core) to write back any data they have 
in modified state at the time of instruction execution and to invalidate their contents. 


Note, non-shared caches may not be written back nor invalidated. In Figure 2-10 below, if code executing on either 
LPO or LP1 were to execute a WBINVD, the shared L1 and L2 for LPO/LP1 will be written back and invalidated as will 
the shared L3. However, the L1 and L2 caches not shared with LPO and LP1 will not be written back nor invalidated. 
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Figure 2-10. WBINVD Invalidation of Shared and Non-Shared Cache Hierarchy 
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The INVLPG (invalidate TLB entry) instruction invalidates (flushes) the TLB entry for a specified page. 


2.8.5 Controlling the Processor 


The HLT (halt processor) instruction stops the processor until an enabled interrupt (such as NMI or SMI, which are 
normally enabled), a debug exception, the BINIT# signal, the INIT# signal, or the RESET# signal is received. The 
processor generates a special bus cycle to indicate that the halt mode has been entered. 


Hardware may respond to this signal in a number of ways. An indicator light on the front panel may be turned on. 
An NMI interrupt for recording diagnostic information may be generated. Reset initialization may be invoked (note 
that the BINIT# pin was introduced with the Pentium Pro processor). If any non-wake events are pending during 

shutdown, they will be handled after the wake event from shutdown is processed (for example, A20M# interrupts). 


The LOCK prefix invokes a locked (atomic) read-modify-write operation when modifying a memory operand. This 
mechanism is used to allow reliable communications between processors in multiprocessor systems, as described 
below: 


¢ Inthe Pentium processor and earlier IA-32 processors, the LOCK prefix causes the processor to assert the 
LOCK# signal during the instruction. This always causes an explicit bus lock to occur. 


¢ Inthe Pentium 4, Intel Xeon, and P6 family processors, the locking operation is handled with either a cache lock 
or bus lock. If a memory access is cacheable and affects only a single cache line, a cache lock is invoked and 
the system bus and the actual memory location in system memory are not locked during the operation. Here, 
other Pentium 4, Intel Xeon, or P6 family processors on the bus write-back any modified data and invalidate 
their caches as necessary to maintain system memory coherency. If the memory access is not cacheable 
and/or it crosses a cache line boundary, the processor’s LOCK# signal is asserted and the processor does not 
respond to requests for bus control during the locked operation. 


The RSM (return from SMM) instruction restores the processor (from a context dump) to the state it was in prior to 
a system management mode (SMM) interrupt. 


2.8.6 Reading Performance-Monitoring and Time-Stamp Counters 


The RDPMC (read performance-monitoring counter) and RDTSC (read time-stamp counter) instructions allow 
application programs to read the processor’s performance-monitoring and time-stamp counters, respectively. 
Processors based on Intel NetBurst® microarchitecture have eighteen 40-bit performance-monitoring counters; P6 
family processors have two 40-bit counters. Intel® Atom™ processors and most of the processors based on the 
Intel Core microarchitecture support two types of performance monitoring counters: programmable performance 
counters similar to those available in the P6 family, and three fixed-function performance monitoring counters. 
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Details of programmable and fixed-function performance monitoring counters for each processor generation are 
described in Chapter 18, “Performance Monitoring”. 


The programmable performance counters can support counting either the occurrence or duration of events. Events 
that can be monitored on programmable counters generally are model specific (except for architectural perfor- 
mance events enumerated by CPUID leaf OAH); they may include the number of instructions decoded, interrupts 
received, or the number of cache loads. Individual counters can be set up to monitor different events. Use the 
system instruction WRMSR to set up values in one of the IA32_PERFEVTSELx MSR, in one of the 45 ESCRs and one 
of the 18 CCCR MSRs (for Pentium 4 and Intel Xeon processors); or in the PerfEvtSelO or the PerfEvtSeli MSR (for 
the P6 family processors). The RDPMC instruction loads the current count from the selected counter into the 
EDX:EAX registers. 


Fixed-function performance counters record only specific events that are defined at: https://perfmon- 
events.intel.com/, and the width/number of fixed-function counters are enumerated by CPUID leaf OAH. 


The time-stamp counter is a model-specific 64-bit counter that is reset to zero each time the processor is reset. If 
not reset, the counter will increment ~9.5 x 101° times per year when the processor is operating at a clock rate 
of 3GHz. At this clock frequency, it would take over 190 years for the counter to wrap around. The RDTSC 
instruction loads the current count of the time-stamp counter into the EDX:EAX registers. 


See Section 18.1, “Performance Monitoring Overview,” and Section 17.17, “Time-Stamp Counter,” for more infor- 
mation about the performance monitoring and time-stamp counters. 


The RDTSC instruction was introduced into the IA-32 architecture with the Pentium processor. The RDPMC instruc- 
tion was introduced into the IA-32 architecture with the Pentium Pro processor and the Pentium processor with 
MMX technology. Earlier Pentium processors have two performance-monitoring counters, but they can be read only 
with the RDMSR instruction, and only at privilege level 0. 


2.8.6.1 Reading Counters in 64-Bit Mode 


In 64-bit mode, RDTSC operates the same as in protected mode. The count in the time-stamp counter is stored in 
EDX:EAX (or RDX[31:0]:RAX[31:0] with RDX[63:32]:RAX[63:32] cleared). 


RDPMC requires an index to specify the offset of the performance-monitoring counter. In 64-bit mode for Pentium 
4 or Intel Xeon processor families, the index is specified in ECX[30:0]. The current count of the performance-moni- 
toring counter is stored in EDX:EAX (or RDX[31:0]:RAX[31:0] with RDX[63:32]:RAX[63:32] cleared). 


2.8.7 Reading and Writing Model-Specific Registers 


The RDMSR (read model-specific register) and WRMSR (write model-specific register) instructions allow a 
processor’s 64-bit model-specific registers (MSRs) to be read and written, respectively. The MSR to be read or 
written is specified by the value in the ECX register. 


RDMSR reads the value from the specified MSR to the EDX:EAX registers; WRMSR writes the value in the EDX:EAX 
registers to the specified MSR. RDMSR and WRMSR were introduced into the IA-32 architecture with the Pentium 
processor. 


See Section 9.4, “Model-Specific Registers (MSRs),” for more information. 


2.8.7.1 Reading and Writing Model-Specific Registers in 64-Bit Mode 


RDMSR and WRMSR require an index to specify the address of an MSR. In 64-bit mode, the index is 32 bits; it is 
specified using ECX. 


2.8.8 Enabling Processor Extended States 


The XSETBV instruction is required to enable OS support of individual processor extended states in XCRO (see 
Section 2.6). 
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CHAPTER 3 
PROTECTED-MODE MEMORY MANAGEMENT 


This chapter describes the Intel 64 and IA-32 architecture’s protected-mode memory management facilities, 
including the physical memory requirements, segmentation mechanism, and paging mechanism. 


See also: Chapter 5, “Protection” (for a description of the processor’s protection mechanism) and Chapter 19, 
“8086 Emulation” (for a description of memory addressing protection in real-address and virtual-8086 modes). 


3.1 MEMORY MANAGEMENT OVERVIEW 


The memory management facilities of the IA-32 architecture are divided into two parts: segmentation and paging. 
Segmentation provides a mechanism of isolating individual code, data, and stack modules so that multiple 
programs (or tasks) can run on the same processor without interfering with one another. Paging provides a mech- 
anism for implementing a conventional demand-paged, virtual-memory system where sections of a program's 
execution environment are mapped into physical memory as needed. Paging can also be used to provide isolation 
between multiple tasks. When operating in protected mode, some form of segmentation must be used. There is 
no mode bit to disable segmentation. The use of paging, however, is optional. 


These two mechanisms (Segmentation and paging) can be configured to support simple single-program (or single- 
task) systems, multitasking systems, or multiple-processor systems that used shared memory. 


As shown in Figure 3-1, segmentation provides a mechanism for dividing the processor’s addressable memory 
space (called the linear address space) into smaller protected address spaces called segments. Segments can 
be used to hold the code, data, and stack for a program or to hold system data structures (such as a TSS or LDT). 
If more than one program (or task) is running on a processor, each program can be assigned its own set of 
segments. The processor then enforces the boundaries between these segments and ensures that one program 
does not interfere with the execution of another program by writing into the other program's segments. The 
segmentation mechanism also allows typing of segments so that the operations that may be performed on a partic- 
ular type of segment can be restricted. 


All the segments in a system are contained in the processor’s linear address space. To locate a byte in a particular 
segment, a logical address (also called a far pointer) must be provided. A logical address consists of a segment 
selector and an offset. The segment selector is a unique identifier for a segment. Among other things it provides an 
offset into a descriptor table (such as the global descriptor table, GDT) to a data structure called a segment 
descriptor. Each segment has a segment descriptor, which specifies the size of the segment, the access rights and 
privilege level for the segment, the segment type, and the location of the first byte of the segment in the linear 
address space (called the base address of the segment). The offset part of the logical address is added to the base 
address for the segment to locate a byte within the segment. The base address plus the offset thus forms a linear 
address in the processor’s linear address space. 
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Figure 3-1. Segmentation and Paging 


If paging is not used, the linear address space of the processor is mapped directly into the physical address space 
of processor. The physical address space is defined as the range of addresses that the processor can generate on 
its address bus. 


Because multitasking computing systems commonly define a linear address space much larger than it is economi- 
cally feasible to contain all at once in physical memory, some method of “virtualizing” the linear address space is 
needed. This virtualization of the linear address space is handled through the processor’s paging mechanism. 


Paging supports a “virtual memory” environment where a large linear address space is simulated with a small 
amount of physical memory (RAM and ROM) and some disk storage. When using paging, each segment is divided 
into pages (typically 4 KBytes each in size), which are stored either in physical memory or on the disk. The oper- 
ating system or executive maintains a page directory and a set of page tables to keep track of the pages. When a 
program (or task) attempts to access an address location in the linear address space, the processor uses the page 
directory and page tables to translate the linear address into a physical address and then performs the requested 
operation (read or write) on the memory location. 


If the page being accessed is not currently in physical memory, the processor interrupts execution of the program 
(by generating a page-fault exception). The operating system or executive then reads the page into physical 
memory from the disk and continues executing the program. 


When paging is implemented properly in the operating-system or executive, the swapping of pages between phys- 
ical memory and the disk is transparent to the correct execution of a program. Even programs written for 16-bit IA- 
32 processors can be paged (transparently) when they are run in virtual-8086 mode. 


3.2 USING SEGMENTS 


The segmentation mechanism supported by the IA-32 architecture can be used to implement a wide variety of 
system designs. These designs range from flat models that make only minimal use of segmentation to protect 
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programs to multi-segmented models that employ segmentation to create a robust operating environment in 
which multiple programs and tasks can be executed reliably. 


The following sections give several examples of how segmentation can be employed in a system to improve 
memory management performance and reliability. 


3.2.1 Basic Flat Model 


The simplest memory model for a system is the basic “flat model,” in which the operating system and application 

programs have access to a continuous, unsegmented address space. To the greatest extent possible, this basic flat 
model hides the segmentation mechanism of the architecture from both the system designer and the application 

programmer. 


To implement a basic flat memory model with the IA-32 architecture, at least two segment descriptors must be 
created, one for referencing a code segment and one for referencing a data segment (see Figure 3-2). Both of 
these segments, however, are mapped to the entire linear address space: that is, both segment descriptors have 
the same base address value of 0 and the same segment limit of 4 GBytes. By setting the segment limit to 4 
GBytes, the segmentation mechanism is kept from generating exceptions for out of limit memory references, even 
if no physical memory resides at a particular address. ROM (EPROM) is generally located at the top of the physical 
address space, because the processor begins execution at FFFF_FFFOH. RAM (DRAM) is placed at the bottom of the 
address space because the initial base address for the DS data segment after reset initialization is 0. 


3.2.2 Protected Flat Model 


The protected flat model is similar to the basic flat model, except the segment limits are set to include only the 
range of addresses for which physical memory actually exists (see Figure 3-3). A general-protection exception 
(#GP) is then generated on any attempt to access nonexistent memory. This model provides a minimum level of 
hardware protection against some kinds of program bugs. 
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Figure 3-2. Flat Model 
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Figure 3-3. Protected Flat Model 


More complexity can be added to this protected flat model to provide more protection. For example, for the paging 
mechanism to provide isolation between user and supervisor code and data, four segments need to be defined: 
code and data segments at privilege level 3 for the user, and code and data segments at privilege level 0 for the 
supervisor. Usually these segments all overlay each other and start at address 0 in the linear address space. This 
flat segmentation model along with a simple paging structure can protect the operating system from applications, 
and by adding a separate paging structure for each task or process, it can also protect applications from each other. 
Similar designs are used by several popular multitasking operating systems. 


3.2.3 Multi-Segment Model 


A multi-segment model (such as the one shown in Figure 3-4) uses the full capabilities of the segmentation mech- 
anism to provide hardware enforced protection of code, data structures, and programs and tasks. Here, each 
program (or task) is given its own table of segment descriptors and its own segments. The segments can be 
completely private to their assigned programs or shared among programs. Access to all segments and to the 
execution environments of individual programs running on the system is controlled by hardware. 
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Figure 3-4. Multi-Segment Model 


Access checks can be used to protect not only against referencing an address outside the limit of a segment, but 
also against performing disallowed operations in certain segments. For example, since code segments are desig- 
nated as read-only segments, hardware can be used to prevent writes into code segments. The access rights infor- 
mation created for segments can also be used to set up protection rings or levels. Protection levels can be used to 
protect operating-system procedures from unauthorized access by application programs. 


3.2.4 Segmentation in |[A-32e Mode 


In IA-32e mode of Intel 64 architecture, the effects of segmentation depend on whether the processor is running 
in compatibility mode or 64-bit mode. In compatibility mode, segmentation functions just as it does using legacy 
16-bit or 32-bit protected mode semantics. 


In 64-bit mode, segmentation is generally (but not completely) disabled, creating a flat 64-bit linear-address 
space. The processor treats the segment base of CS, DS, ES, SS as zero, creating a linear address that is equal to 
the effective address. The FS and GS segments are exceptions. These segment registers (which hold the segment 
base) can be used as additional base registers in linear address calculations. They facilitate addressing local data 
and certain operating system data structures. 


Note that the processor does not perform segment limit checks at runtime in 64-bit mode. 


3.2.5 Paging and Segmentation 


Paging can be used with any of the segmentation models described in Figures 3-2, 3-3, and 3-4. The processor’s 
paging mechanism divides the linear address space (into which segments are mapped) into pages (as shown in 
Figure 3-1). These linear-address-space pages are then mapped to pages in the physical address space. The 
paging mechanism offers several page-level protection facilities that can be used with or instead of the segment- 
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protection facilities. For example, it lets read-write protection be enforced on a page-by-page basis. The paging 
mechanism also provides two-level user-supervisor protection that can also be specified on a page-by-page basis. 


3.3 PHYSICAL ADDRESS SPACE 


In protected mode, the IA-32 architecture provides a normal physical address space of 4 GBytes (23bytes). This 
is the address space that the processor can address on its address bus. This address space is flat (unsegmented), 
with addresses ranging continuously from 0 to FFFFFFFFH. This physical address space can be mapped to read- 
write memory, read-only memory, and memory mapped I/O. The memory mapping facilities described in this 
chapter can be used to divide this physical memory up into segments and/or pages. 


Starting with the Pentium Pro processor, the IA-32 architecture also supports an extension of the physical address 
space to 276 bytes (64 GBytes); with a maximum physical address of FFFFFFFFFH. This extension is invoked in 
either of two ways: 


e Using the physical address extension (PAE) flag, located in bit 5 of control register CR4. 
e Using the 36-bit page size extension (PSE-36) feature (introduced in the Pentium IIl processors). 


Physical address support has since been extended beyond 36 bits. See Chapter 4, “Paging” for more information 
about 36-bit physical addressing. 


3.3.1 Intel® 64 Processors and Physical Address Space 


On processors that support Intel 64 architecture (CPUID.80000001H:EDX[29] = 1), the size of the physical 
address range is implementation-specific and indicated by CPUID.80000008H:EAX[bits 7-0]. 


For the format of information returned in EAX, see “CPUID—CPU Identification” in Chapter 3 of the Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 2A. See also: Chapter 4, “Paging.” 


3.4 LOGICAL AND LINEAR ADDRESSES 


At the system-architecture level in protected mode, the processor uses two stages of address translation to arrive 
at a physical address: logical-address translation and linear address space paging. 


Even with the minimum use of segments, every byte in the processor’s address space is accessed with a logical 
address. A logical address consists of a 16-bit segment selector and a 32-bit offset (see Figure 3-5). The segment 
selector identifies the segment the byte is located in and the offset specifies the location of the byte in the segment 
relative to the base address of the segment. 


The processor translates every logical address into a linear address. A linear address is a 32-bit address in the 
processor's linear address space. Like the physical address space, the linear address space is a flat (unsegmented), 
232-byte address space, with addresses ranging from 0 to FFFFFFFFH. The linear address space contains all the 
segments and system tables defined for a system. 


To translate a logical address into a linear address, the processor does the following: 


1. Uses the offset in the segment selector to locate the segment descriptor for the segment in the GDT or LDT and 
reads it into the processor. (This step is needed only when a new segment selector is loaded into a segment 
register. ) 


2. Examines the segment descriptor to check the access rights and range of the segment to ensure that the 
segment is accessible and that the offset is within the limits of the segment. 


3. Adds the base address of the segment from the segment descriptor to the offset to form a linear address. 
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Figure 3-5. Logical Address to Linear Address Translation 


If paging is not used, the processor maps the linear address directly to a physical address (that is, the linear 
address goes out on the processor’s address bus). If the linear address space is paged, a second level of address 
translation is used to translate the linear address into a physical address. 


See also: Chapter 4, “Paging.” 


3.4.1 Logical Address Translation in |A-32e Mode 


In IA-32e mode, an Intel 64 processor uses the steps described above to translate a logical address to a linear 
address. In 64-bit mode, the offset and base address of the segment are 64-bits instead of 32 bits. The linear 
address format is also 64 bits wide and is subject to the canonical form requirement. 


Each code segment descriptor provides an L bit. This bit allows a code segment to execute 64-bit code or legacy 
32-bit code by code segment. 


3.4.2 Segment Selectors 


A segment selector is a 16-bit identifier for a segment (see Figure 3-6). It does not point directly to the segment, 
but instead points to the segment descriptor that defines the segment. A segment selector contains the following 
items: 


Index (Bits 3 through 15) — Selects one of 8192 descriptors in the GDT or LDT. The processor multiplies 
the index value by 8 (the number of bytes in a segment descriptor) and adds the result to the base 
address of the GDT or LDT (from the GDTR or LDTR register, respectively). 


TI (table indicator) flag 
(Bit 2) — Specifies the descriptor table to use: clearing this flag selects the GDT; setting this flag 
selects the current LDT. 


Index 


Table Indicator 
0 = GDT 
1 =LDT 
Requested Privilege Level (RPL) 


Figure 3-6. Segment Selector 
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Requested Privilege Level (RPL) 
(Bits 0 and 1) — Specifies the privilege level of the selector. The privilege level can range from 0 to 
3, with 0 being the most privileged level. See Section 5.5, “Privilege Levels”, for a description of the 
relationship of the RPL to the CPL of the executing program (or task) and the descriptor privilege 
level (DPL) of the descriptor the segment selector points to. 


The first entry of the GDT is not used by the processor. A segment selector that points to this entry of the GDT (that 
is, asegment selector with an index of 0 and the TI flag set to 0) is used as a “null segment selector.” The processor 
does not generate an exception when a segment register (other than the CS or SS registers) is loaded with a null 
selector. It does, however, generate an exception when a segment register holding a null selector is used to access 
memory. A null selector can be used to initialize unused segment registers. Loading the CS or SS register with a null 
segment selector causes a general-protection exception (#GP) to be generated. 


Segment selectors are visible to application programs as part of a pointer variable, but the values of selectors are 
usually assigned or modified by link editors or linking loaders, not application programs. 


3.4.3 Segment Registers 


To reduce address translation time and coding complexity, the processor provides registers for holding up to 6 
segment selectors (see Figure 3-7). Each of these segment registers support a specific kind of memory reference 
(code, stack, or data). For virtually any kind of program execution to take place, at least the code-segment (CS), 
data-segment (DS), and stack-segment (SS) registers must be loaded with valid segment selectors. The processor 
also provides three additional data-segment registers (ES, FS, and GS), which can be used to make additional data 
segments available to the currently executing program (or task). 


For a program to access a segment, the segment selector for the segment must have been loaded in one of the 
segment registers. So, although a system can define thousands of segments, only 6 can be available for immediate 
use. Other segments can be made available by loading their segment selectors into these registers during program 
execution. 


Visible Part Hidden Part 
Segment Selector Base Address, Limit, Access Information | CS 
Ss 
DS 
ES 
FS 
GS 


Figure 3-7. Segment Registers 


Every segment register has a “visible” part and a “hidden” part. (The hidden part is sometimes referred to as a 
“descriptor cache” or a “shadow register.”) When a segment selector is loaded into the visible part of a segment 
register, the processor also loads the hidden part of the segment register with the base address, segment limit, and 
access control information from the segment descriptor pointed to by the segment selector. The information cached 
in the segment register (visible and hidden) allows the processor to translate addresses without taking extra bus 
cycles to read the base address and limit from the segment descriptor. In systems in which multiple processors 
have access to the same descriptor tables, it is the responsibility of software to reload the segment registers when 
the descriptor tables are modified. If this is not done, an old segment descriptor cached in a segment register might 
be used after its memory-resident version has been modified. 


Two kinds of load instructions are provided for loading the segment registers: 


1. Direct load instructions such as the MOV, POP, LDS, LES, LSS, LGS, and LFS instructions. These instructions 
explicitly reference the segment registers. 


2. Implied load instructions such as the far pointer versions of the CALL, JMP, and RET instructions, the SYSENTER 
and SYSEXIT instructions, and the IRET, INT n, INTO, INT3, and INT1 instructions. These instructions change 
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the contents of the CS register (and sometimes other segment registers) as an incidental part of their 
operation. 


The MOV instruction can also be used to store the visible part of a segment register in a general-purpose register. 


3.4.4 Segment Loading Instructions in |[A-32e Mode 


Because ES, DS, and SS segment registers are not used in 64-bit mode, their fields (base, limit, and attribute) in 
segment descriptor registers are ignored. Some forms of segment load instructions are also invalid (for example, 
LDS, POP ES). Address calculations that reference the ES, DS, or SS segments are treated as if the segment base 
is zero. 


The processor checks that all linear-address references are in canonical form instead of performing limit checks. 
Mode switching does not change the contents of the segment registers or the associated descriptor registers. 
These registers are also not changed during 64-bit mode execution, unless explicit segment loads are performed. 


In order to set up compatibility mode for an application, segment-load instructions (MOV to Sreg, POP Sreg) work 
normally in 64-bit mode. An entry is read from the system descriptor table (GDT or LDT) and is loaded in the hidden 
portion of the segment register. The descriptor-register base, limit, and attribute fields are all loaded. However, the 
contents of the data and stack segment selector and the descriptor registers are ignored. 


When FS and GS segment overrides are used in 64-bit mode, their respective base addresses are used in the linear 
address calculation: (FS or GS).base + index + displacement. FS.base and GS.base are then expanded to the full 
linear-address size supported by the implementation. The resulting effective address calculation can wrap across 
positive and negative addresses; the resulting linear address must be canonical. 


In 64-bit mode, memory accesses using FS-segment and GS-segment overrides are not checked for a runtime limit 
nor subjected to attribute-checking. Normal segment loads (MOV to Sreg and POP Sreg) into FS and GS load a 
standard 32-bit base value in the hidden portion of the segment register. The base address bits above the standard 
32 bits are cleared to 0 to allow consistency for implementations that use less than 64 bits. 


The hidden descriptor register fields for FS.base and GS.base are physically mapped to MSRs in order to load all 
address bits supported by a 64-bit implementation. Software with CPL = 0 (privileged software) can load all 
supported linear-address bits into FS.base or GS.base using WRMSR. Addresses written into the 64-bit FS.base 
and GS.base registers must be in canonical form. A WRMSR instruction that attempts to write a non-canonical 
address to those registers causes a #GP fault. 


When in compatibility mode, FS and GS overrides operate as defined by 32-bit mode behavior regardless of the 
value loaded into the upper 32 linear-address bits of the hidden descriptor register base field. Compatibility mode 
ignores the upper 32 bits when calculating an effective address. 


A new 64-bit mode instruction, SWAPGS, can be used to load GS base. SWAPGS exchanges the kernel data struc- 
ture pointer from the IA32_KERNEL_GS_BASE MSR with the GS base register. The kernel can then use the GS 
prefix on normal memory references to access the kernel data structures. An attempt to write a non-canonical 
value (using WRMSR) to the IA32_KERNEL_GS_BASE MSR causes a #GP fault. 


3.4.5 Segment Descriptors 


A segment descriptor is a data structure in a GDT or LDT that provides the processor with the size and location of 
a segment, as well as access control and status information. Segment descriptors are typically created by 
compilers, linkers, loaders, or the operating system or executive, but not application programs. Figure 3-8 illus- 
trates the general descriptor format for all types of segment descriptors. 
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BASE — Segment base address 
D/B — Default operation size (0 = 16-bit segment; 1 = 32-bit segment) 
DPL — Descriptor privilege level 
G — Granularity 
LIMIT — Segment Limit 
P — Segment present 
S) — Descriptor type (0 = system; 1 = code or data) 
TYPE — Segment type 


Figure 3-8. Segment Descriptor 


The flags and fields in a segment descriptor are as follows: 


Segment limit field 


Specifies the size of the segment. The processor puts together the two segment limit fields to form 

a 20-bit value. The processor interprets the segment limit in one of two ways, depending on the 

setting of the G (granularity) flag: 

e If the granularity flag is clear, the segment size can range from 1 byte to 1 MByte, in byte incre- 
ments. 

e If the granularity flag is set, the segment size can range from 4 KBytes to 4 GBytes, in 4-KByte 
increments. 


The processor uses the segment limit in two different ways, depending on whether the segment is 
an expand-up or an expand-down segment. See Section 3.4.5.1, “Code- and Data-Segment 
Descriptor Types”, for more information about segment types. For expand-up segments, the offset 
in a logical address can range from 0 to the segment limit. Offsets greater than the segment limit 
generate general-protection exceptions (#GP, for all segments other than SS) or stack-fault excep- 
tions (#SS for the SS segment). For expand-down segments, the segment limit has the reverse 
function; the offset can range from the segment limit plus 1 to FFFFFFFFH or FFFFH, depending on 
the setting of the B flag. Offsets less than or equal to the segment limit generate general-protection 
exceptions or stack-fault exceptions. Decreasing the value in the segment limit field for an expand- 
down segment allocates new memory at the bottom of the segment's address space, rather than at 
the top. IA-32 architecture stacks always grow downwards, making this mechanism convenient for 


expandable stacks. 


Base address fields 


Type field 
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Defines the location of byte 0 of the segment within the 4-GByte linear address space. The 
processor puts together the three base address fields to form a single 32-bit value. Segment base 
addresses should be aligned to 16-byte boundaries. Although 16-byte alignment is not required, 
this alignment allows programs to maximize performance by aligning code and data on 16-byte 
boundaries. 


Indicates the segment or gate type and specifies the kinds of access that can be made to the 
segment and the direction of growth. The interpretation of this field depends on whether the 
descriptor type flag specifies an application (code or data) descriptor or a system descriptor. The 
encoding of the type field is different for code, data, and system descriptors (see Figure 5-1). See 
Section 3.4.5.1, “Code- and Data-Segment Descriptor Types”, for a description of how this field is 
used to specify code and data-segment types. 
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S (descriptor type) flag 
Specifies whether the segment descriptor is for a system segment (S flag is clear) or a code or data 
segment (S flag is set). 


DPL (descriptor privilege level) field 
Specifies the privilege level of the segment. The privilege level can range from 0 to 3, with 0 being 
the most privileged level. The DPL is used to control access to the segment. See Section 5.5, “Priv- 
ilege Levels”, for a description of the relationship of the DPL to the CPL of the executing code 
segment and the RPL of a segment selector. 


P (segment-present) flag 
Indicates whether the segment is present in memory (set) or not present (clear). If this flag is clear, 
the processor generates a segment-not-present exception (#NP) when a segment selector that 
points to the segment descriptor is loaded into a segment register. Memory management software 
can use this flag to control which segments are actually loaded into physical memory at a given 
time. It offers a control in addition to paging for managing virtual memory. 


Figure 3-9 shows the format of a segment descriptor when the segment-present flag is clear. When 
this flag is clear, the operating system or executive is free to use the locations marked “Available” to 
store its own data, such as information regarding the whereabouts of the missing segment. 


D/B (default operation size/default stack pointer size and/or upper bound) flag 
Performs different functions depending on whether the segment descriptor is an executable code 
segment, an expand-down data segment, or a stack segment. (This flag should always be set to 1 
for 32-bit code and data segments and to 0 for 16-bit code and data segments. ) 


Executable code segment. The flag is called the D flag and it indicates the default length for 
effective addresses and operands referenced by instructions in the segment. If the flag is set, 
32-bit addresses and 32-bit or 8-bit operands are assumed; if it is clear, 16-bit addresses and 
16-bit or 8-bit operands are assumed. 

The instruction prefix 66H can be used to select an operand size other than the default, and the 
prefix 67H can be used select an address size other than the default. 


Stack segment (data segment pointed to by the SS register). The flag is called the B (big) 
flag and it specifies the size of the stack pointer used for implicit stack operations (such as 
pushes, pops, and calls). If the flag is set, a 32-bit stack pointer is used, which is stored in the 
32-bit ESP register; if the flag is clear, a 16-bit stack pointer is used, which is stored in the 16- 
bit SP register. If the stack segment is set up to be an expand-down data segment (described in 
the next paragraph), the B flag also specifies the upper bound of the stack segment. 


Expand-down data segment. The flag is called the B flag and it specifies the upper bound of 
the segment. If the flag is set, the upper bound is FFFFFFFFH (4 GBytes); if the flag is clear, the 
upper bound is FFFFH (64 KBytes). 


31 16 15 14 1312 11 87 0 
D . 
Available 0} P |S; Type Available 4 
31 0 
Available 0 


Figure 3-9. Segment Descriptor When Segment-Present Flag Is Clear 


G (granularity) flag 
Determines the scaling of the segment limit field. When the granularity flag is clear, the segment 
limit is interpreted in byte units; when flag is set, the segment limit is interpreted in 4-KByte units. 
(This flag does not affect the granularity of the base address; it is always byte granular.) When the 
granularity flag is set, the twelve least significant bits of an offset are not tested when checking the 
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offset against the segment limit. For example, when the granularity flag is set, a limit of 0 results in 
valid offsets from 0 to 4095. 


L (64-bit code segment) flag 
In IA-32e mode, bit 21 of the second doubleword of the segment descriptor indicates whether a 
code segment contains native 64-bit code. A value of 1 indicates instructions in this code segment 
are executed in 64-bit mode. A value of 0 indicates the instructions in this code segment are 
executed in compatibility mode. If L-bit is set, then D-bit must be cleared. When not in IA-32e mode 
or for non-code segments, bit 21 is reserved and should always be set to 0. 


Available and reserved bits 
Bit 20 of the second doubleword of the segment descriptor is available for use by system software. 


3.4.5.1 Code- and Data-Segment Descriptor Types 


When the S (descriptor type) flag in a segment descriptor is set, the descriptor is for either a code or a data 
segment. The highest order bit of the type field (bit 11 of the second double word of the segment descriptor) then 
determines whether the descriptor is for a data segment (clear) or a code segment (set). 


For data segments, the three low-order bits of the type field (bits 8, 9, and 10) are interpreted as accessed (A), 
write-enable (W), and expansion-direction (E). See Table 3-1 for a description of the encoding of the bits in the 
type field for code and data segments. Data segments can be read-only or read/write segments, depending on the 
setting of the write-enable bit. 


Table 3-1. Code- and Data-Segment Types 


Type Field Descriptor Description 
Decimal 1 10 9 8 Type 
E W A 
0 0 0 0 0 Data Read-Only 
1 0 0 0 1 Data Read-Only, accessed 
2 0 0 1 0 Data Read/Write 
3 0 0 1 1 Data Read/Write, accessed 
4 0 1 0 0 Data Read-Only, expand-down 
5 0 1 0 1 Data Read-Only, expand-down, accessed 
6 0 1 1 0 Data Read/Write, expand-down 
7 0 1 1 1 Data Read/Write, expand-down, accessed 
C R A 
8 1 0 0 0 Code Execute-Only 
9 1 0 0 1 Code Execute-Only, accessed 
10 1 0 1 0 Code Execute/Read 
11 1 0 1 1 Code Execute/Read, accessed 
12 1 1 0 0 Code Execute-Only, conforming 
13 1 1 0 1 Code Execute-Only, conforming, accessed 
14 1 1 1 0 Code Execute/Read, conforming 
15 1 1 1 1 Code Execute/Read, conforming, accessed 


Stack segments are data segments which must be read/write segments. Loading the SS register with a segment 
selector for a nonwritable data segment generates a general-protection exception (#GP). If the size of a stack 
segment needs to be changed dynamically, the stack segment can be an expand-down data segment (expansion- 
direction flag set). Here, dynamically changing the segment limit causes stack space to be added to the bottom of 
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the stack. If the size of a stack segment is intended to remain static, the stack segment may be either an expand- 
up or expand-down type. 


The accessed bit indicates whether the segment has been accessed since the last time the operating-system or 
executive cleared the bit. The processor sets this bit whenever it loads a segment selector for the segment into a 
segment register, assuming that the type of memory that contains the segment descriptor supports processor 
writes. The bit remains set until explicitly cleared. This bit can be used both for virtual memory management and 
for debugging. 


For code segments, the three low-order bits of the type field are interpreted as accessed (A), read enable (R), and 
conforming (C). Code segments can be execute-only or execute/read, depending on the setting of the read-enable 
bit. An execute/read segment might be used when constants or other static data have been placed with instruction 
code in a ROM. Here, data can be read from the code segment either by using an instruction with a CS override 
prefix or by loading a segment selector for the code segment in a data-segment register (the DS, ES, FS, or GS 
registers). In protected mode, code segments are not writable. 


Code segments can be either conforming or nonconforming. A transfer of execution into a more-privileged 
conforming segment allows execution to continue at the current privilege level. A transfer into a nonconforming 
segment at a different privilege level results in a general-protection exception (#GP), unless a call gate or task gate 
is used (see Section 5.8.1, “Direct Calls or Jumps to Code Segments”, for more information on conforming and 
nonconforming code segments). System utilities that do not access protected facilities and handlers for some types 
of exceptions (such as, divide error or overflow) may be loaded in conforming code segments. Utilities that need to 
be protected from less privileged programs and procedures should be placed in nonconforming code segments. 


NOTE 


Execution cannot be transferred by a call or a jump to a less-privileged (numerically higher 
privilege level) code segment, regardless of whether the target segment is a conforming or 
nonconforming code segment. Attempting such an execution transfer will result in a general- 
protection exception. 


All data segments are nonconforming, meaning that they cannot be accessed by less privileged programs or proce- 
dures (code executing at numerically higher privilege levels). Unlike code segments, however, data segments can 
be accessed by more privileged programs or procedures (code executing at numerically lower privilege levels) 
without using a special access gate. 


If the segment descriptors in the GDT or an LDT are placed in ROM, the processor can enter an indefinite loop if 
software or the processor attempts to update (write to) the ROM-based segment descriptors. To prevent this 
problem, set the accessed bits for all segment descriptors placed in a ROM. Also, remove operating-system or 
executive code that attempts to modify segment descriptors located in ROM. 


3.5 SYSTEM DESCRIPTOR TYPES 


When the S (descriptor type) flag in a segment descriptor is clear, the descriptor type is a system descriptor. The 
processor recognizes the following types of system descriptors: 


¢ Local descriptor-table (LDT) segment descriptor. 
° Task-state segment (TSS) descriptor. 

* Call-gate descriptor. 

¢  Interrupt-gate descriptor. 

° Trap-gate descriptor. 

¢ Task-gate descriptor. 


These descriptor types fall into two categories: system-segment descriptors and gate descriptors. System- 
segment descriptors point to system segments (LDT and TSS segments). Gate descriptors are in themselves 
“gates,” which hold pointers to procedure entry points in code segments (call, interrupt, and trap gates) or which 
hold segment selectors for TSS’s (task gates). 
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Table 3-2 shows the encoding of the type field for system-segment descriptors and gate descriptors. Note that 


system descriptors in IA-32e mode are 16 bytes instead of 8 bytes. 


Table 3-2. System-Segment and Gate-Descriptor Types 


Type Field Description 
Decimal 11 10 9 8 32-Bit Mode IA-32e Mode 

0 0 0 0 0 Reserved Reserved 

1 0 0 0 1 16-bit TSS (Available) Reserved 

2 0 0 1 0 LDT LDT 

3 0 0 1 1 16-bit TSS (Busy) Reserved 

4 0 1 0 0 16-bit Call Gate Reserved 

2) 0 1 0 1 Task Gate Reserved 

6 0 1 1 0 16-bit Interrupt Gate Reserved 

7 0 1 1 1 16-bit Trap Gate Reserved 

8 1 0 0 0 Reserved Reserved 

9 1 0 0 1 32-bit TSS (Available) 64-bit TSS (Available) 
10 1 0 1 0 Reserved Reserved 
11 1 0 1 1 32-bit TSS (Busy) 64-bit TSS (Busy) 
12 1 1 0 0 32-bit Call Gate 64-bit Call Gate 
13 1 1 0 1 Reserved Reserved 
14 1 1 1 0 32-bit Interrupt Gate 64-bit Interrupt Gate 
15 1 1 1 1 32-bit Trap Gate 64-bit Trap Gate 


See also: Section 3.5.1, “Segment Descriptor Tables”, and Section 7.2.2, “TSS Descriptor” (for more information 
on the system-segment descriptors); see Section 5.8.3, “Call Gates”, Section 6.11, “IDT Descriptors”, and Section 
7.2.5, “Task-Gate Descriptor” (for more information on the gate descriptors). 


3.5.1 Segment Descriptor Tables 


A segment descriptor table is an arra 
length and can contain up to 8192 (2 


¢ The global descriptor table (GDT) 
¢ The local descriptor tables (LDT) 
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Global Local 
Descriptor Descriptor 
Table (GDT) Table (LDT) 
Pt = 
| Tl=0 Tl=1 
Segment 
Selector 
56 56 
48 48 
40 40 
32 32 
24 24 
16 16 
8 8 
First Descriptor in 
GDT is Not Used 0 0 
GDTR Register LDTR Register 
Limit Limit 
Base Address Base Address 
Seg. Sel. 


Figure 3-10. Global and Local Descriptor Tables 


Each system must have one GDT defined, which may be used for all programs and tasks in the system. Optionally, 
one or more LDTs can be defined. For example, an LDT can be defined for each separate task being run, or some or 
all tasks can share the same LDT. 


The GDT is not a segment itself; instead, it is a data structure in linear address space. The base linear address and 
limit of the GDT must be loaded into the GDTR register (see Section 2.4, “Memory-Management Registers”). The 

base address of the GDT should be aligned on an eight-byte boundary to yield the best processor performance. The 
limit value for the GDT is expressed in bytes. As with segments, the limit value is added to the base address to get 
the address of the last valid byte. A limit value of 0 results in exactly one valid byte. Because segment descriptors 
are always 8 bytes long, the GDT limit should always be one less than an integral multiple of eight (that is, 8N - 1). 


The first descriptor in the GDT is not used by the processor. A segment selector to this “null descriptor” does not 
generate an exception when loaded into a data-segment register (DS, ES, FS, or GS), but it always generates a 
general-protection exception (#GP) when an attempt is made to access memory using the descriptor. By initializing 
the segment registers with this segment selector, accidental reference to unused segment registers can be guar- 
anteed to generate an exception. 


The LDT is located in a system segment of the LDT type. The GDT must contain a segment descriptor for the LDT 
segment. If the system supports multiple LDTs, each must have a separate segment selector and segment 
descriptor in the GDT. The segment descriptor for an LDT can be located anywhere in the GDT. See Section 3.5, 
“System Descriptor Types”, for information on the LDT segment-descriptor type. 


An LDT is accessed with its segment selector. To eliminate address translations when accessing the LDT, the 
segment selector, base linear address, limit, and access rights of the LDT are stored in the LDTR register (see 
Section 2.4, “Memory-Management Registers”). 


When the GDTR register is stored (using the SGDT instruction), a 48-bit “pseudo-descriptor” is stored in memory 
(see top diagram in Figure 3-11). To avoid alignment check faults in user mode (privilege level 3), the pseudo- 
descriptor should be located at an odd word address (that is, address MOD 4 is equal to 2). This causes the 
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processor to store an aligned word, followed by an aligned doubleword. User-mode programs normally do not store 
pseudo-descriptors, but the possibility of generating an alignment check fault can be avoided by aligning pseudo- 
descriptors in this way. The same alignment should be used when storing the IDTR register using the SIDT instruc- 
tion. When storing the LDTR or task register (using the SLDT or STR instruction, respectively), the pseudo- 
descriptor should be located at a doubleword address (that is, address MOD 4 is equal to 0). 


47 16 15 0 
32-bit Base Address Limit 


79 16 15 0 
64-bit Base Address Limit 


Figure 3-11. Pseudo-Descriptor Formats 


3.5.2 Segment Descriptor Tables in |A-32e Mode 


In IA-32e mode, a segment descriptor table can contain up to 8192 (213) 8-byte descriptors. An entry in the 
segment descriptor table can be 8 bytes. System descriptors are expanded to 16 bytes (occupying the space of two 
entries). 

GDTR and LDTR registers are expanded to hold 64-bit base address. The corresponding pseudo-descriptor is 80 
bits. (see the bottom diagram in Figure 3-11). 


The following system descriptors expand to 16 bytes: 
— Call gate descriptors (see Section 5.8.3.1, “IA-32e Mode Call Gates”) 
— IDT gate descriptors (see Section 6.14.1, “64-Bit Mode IDT”) 
— LDT and TSS descriptors (see Section 7.2.3, “TSS Descriptor in 64-bit mode”). 
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CHAPTER 4 
PAGING 


Chapter 3 explains how segmentation converts logical addresses to linear addresses. Paging (or linear-address 
translation) is the process of translating linear addresses so that they can be used to access memory or I/O 
devices. Paging translates each linear address to a physical address and determines, for each translation, what 
accesses to the linear address are allowed (the address’s access rights) and the type of caching used for such 
accesses (the address’s memory type). 


Intel-64 processors support four different paging modes. These modes are identified and defined in Section 4.1. 
Section 4.2 gives an overview of the translation mechanism that is used in all modes. Section 4.3, Section 4.4, and 
Section 4.5 discuss the four paging modes in detail. 


Section 4.6 details how paging determines and uses access rights. Section 4.7 discusses exceptions that may be 
generated by paging (page-fault exceptions). Section 4.8 considers data which the processor writes in response to 
linear-address accesses (accessed and dirty flags). 


Section 4.9 describes how paging determines the memory types used for accesses to linear addresses. Section 
4.10 provides details of how a processor may cache information about linear-address translation. Section 4.11 
outlines interactions between paging and certain VMX features. Section 4.12 gives an overview of how paging can 
be used to implement virtual memory. 


4.1 PAGING MODES AND CONTROL BITS 


Paging behavior is controlled by the following control bits: 

¢ The WP and PG flags in control register CRO (bit 16 and bit 31, respectively). 

° The PSE, PAE, PGE, LA57, PCIDE, SMEP, SMAP, PKE, CET, and PKS flags in control register CR4 (bit 4, bit 5, 
bit 7, bit 12, bit 17, bit 20, bit 21, bit 22, bit 23, and bit 24, respectively). 

¢ The LME and NXE flags in the IA32_EFER MSR (bit 8 and bit 11, respectively). 

¢ The AC flag in the EFLAGS register (bit 18). 


Software enables paging by using the MOV to CRO instruction to set CRO.PG. Before doing so, software should 
ensure that control register CR3 contains the physical address of the first paging structure that the processor will 
use for linear-address translation (see Section 4.2) and that that structure is initialized as desired. See Table 4-3, 
Table 4-7, and Table 4-12 for the use of CR3 in the different paging modes. 


Section 4.1.1 describes how the values of CRO.PG, CR4.PAE, CR4.LA57, and IA32_EFER.LME determine whether 
paging is enabled and, if so, which of four paging modes is in use. Section 4.1.2 explains how to manage these bits 
to establish or make changes in paging modes. Section 4.1.3 discusses how CRO.WP, CR4.PSE, CR4.PGE, 
CR4.PCIDE, CR4.SMEP, CR4.SMAP, CR4.PKE, CR4.CET, CR4.PKS, and IA32_EFER.NXE modify the operation of the 
different paging modes. 


4.1.1 Four Paging Modes 


If CRO.PG = 0, paging is not used. The logical processor treats all linear addresses as if they were physical 
addresses. CR4.PAE, CR4.LA57, and IA32_EFER.LME are ignored by the processor, as are CRO.WP, CR4.PSE, 
CR4.PGE, CR4.SMEP, CR4.SMAP, and IA32_EFER.NXE. (CR4.CET is also ignored insofar as it affects linear-address 
access rights.) 


Paging is enabled if CRO.PG = 1. Paging can be enabled only if protection is enabled (CRO.PE = 1). If paging is 

enabled, one of four paging modes is used. The values of CR4.PAE, CR4.LA57, and IA32_EFER.LME determine 

which paging mode is used: 

° If CR4.PAE = 0, 32-bit paging is used. 32-bit paging is detailed in Section 4.3. 32-bit paging uses CRO.WP, 
CR4.PSE, CR4.PGE, CR4.SMEP, CR4.SMAP, and CR4.CET as described in Section 4.1.3 and Section 4.6. 
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° If CR4.PAE = 1 and IA32_EFER.LME = 0, PAE paging is used. PAE paging is detailed in Section 4.4. PAE paging 
uses CRO.WP, CR4.PGE, CR4.SMEP, CR4.SMAP, CR4.CET, and IA32_EFER.NXE as described in Section 4.1.3 and 
Section 4.6. 


© If CR4.PAE = 1, IA32_EFER.LME = 1, and CR4.LA57 = 0, 4-level paging! is used. 4-level paging is detailed 
in Section 4.5 (along with 5-level paging). 4-level paging uses CRO.WP, CR4.PGE, CR4.PCIDE, CR4.SMEP, 
CR4.SMAP, CR4.PKE, CR4.CET, CR4.PKS, and IA32_EFER.NXE as described in Section 4.1.3 and Section 4.6. 


© If CR4.PAE = 1, IA32_EFER.LME = 1, and CR4.LA57 = 1, 5-level paging is used. 5-level paging is detailed in 
Section 4.5 (along with 4-level paging). 5-level paging uses CRO.WP, CR4.PGE, CR4.PCIDE, CR4.SMEP, 
CR4.SMAP, CR4.PKE, CR4.CET, CR4.PKS, and IA32_EFER.NXE as described in Section 4.1.3 and Section 4.6. 


NOTE 


32-bit paging and PAE paging can be used only in legacy protected mode (IA32_EFER.LME = 0). In 
contrast, 4-level paging and 5-level paging can be used only IA-32e mode (IA32_EFER.LME = 1). 


The four paging modes differ with regard to the following details: 
¢ Linear-address width. The size of the linear addresses that can be translated. 
¢  Physical-address width. The size of the physical addresses produced by paging. 


¢ Page size. The granularity at which linear addresses are translated. Linear addresses on the same page are 
translated to corresponding physical addresses on the same page. 


* Support for execute-disable access rights. In some paging modes, software can be prevented from fetching 
instructions from pages that are otherwise readable. 


¢ Support for PCIDs. With 4-level paging and 5-level paging, software can enable a facility by which a logical 
processor caches information for multiple linear-address spaces. The processor may retain cached information 
when software switches between different linear-address spaces. 


® Support for protection keys. With 4-level paging and 5-level paging, each linear address is associated with a 
protection key. Software can use the protection-key rights registers to disable, for each protection key, how 
certain accesses to linear addresses associated with that protection key. 


Table 4-1 illustrates the principal differences between the four paging modes. 


Table 4-1. Properties of Different Paging Modes 


a ; Supports 

Paging PGin | PAcin | LMEin ASTI |e eee Page pupports | PCIDs and 
Mode CRO CR4 IA32_EFER | CR4 Width Width! Sizes Disable? aa 
None 0 N/A N/A N/A 32 32 N/A No No 
32-bit 1 ) 02 N/A 32 Up to 402 . eee No No 

4MB 

4KB 5 
PAE 1 1 0 N/A 32 Up to 52 ap Yes No 

4KB 
4-level 1 1 1 0) 48 Up to 52 2 MB Yes? Yes’ 

1 GB® 

4KB 
5-level 1 1 1 1 57 Up to 52 2MB Yes? Yes’ 

1 GB® 


1. Earlier versions of this manual used the term “IA-32e paging” to identify 4-level paging. 


2. The LMA flag in the IA32_EFER MSR (bit 10) is a status bit that indicates whether the logical processor is in IA-32e mode (and thus 
uses either 4-level paging or 5-level paging). The processor always sets IA32_EFER.LMA to CRO.PG & IA32_EFER.LME. Software can- 
not directly modify [A32_EFER.LMA; an execution of WRMSR to the [A32_EFER MSR ignores bit 10 of its source operand. 
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NOTES: 

1. The physical-address width is always bounded by MAXPHYADDR; see Section 4.1.4. 

2. The processor ensures that [A32_EFER.LME must be O if CRO.PG = 1 and CR4.PAE = 0. 

3. 32-bit paging supports physical-address widths of more than 32 bits only for 4-MByte pages and only if the PSE-36 mechanism is 
supported; see Section 4.1.4 and Section 4.3. 

4. 32-bit paging uses 4-MByte pages only if CR4.PSE = 1; see Section 4.3. 

5. Execute-disable access rights are applied only if IA32_EFER.NXE = 1; see Section 4.6. 

6. Processors that support 4-level paging or 5-level paging do not necessarily support 1-GByte pages; see Section 4.1.4. 

7.PCIDs are used only if CR4.PCIDE = 1; see Section 4.10.1. Protection keys are used only if certain conditions hold; see Section 4.6.2. 


Because 32-bit paging and PAE paging are used only in legacy protected mode and because legacy protected mode 
cannot produce linear addresses larger than 32 bits, 32-bit paging and PAE paging translate 32-bit linear 
addresses. 


4-level paging and 5-level paging are used only in IA-32e mode. IA-32e mode has two sub-modes: 


° Compatibility mode. This sub-mode uses only 32-bit linear addresses. In this sub-mode, 4-level paging and 5- 
level paging treat bits 63:32 of such an address as all 0. 


° 64-bit mode. While this sub-mode produces 64-bit linear addresses, the processor enforces canonicality, 
meaning that the upper bits of such an address are identical: bits 63:47 for 4-level paging and bits 63:56 for 
5-level paging. 4-level paging (respectively, 5-level paging) does not use bits 63:48 (respectively, bits 63:57) 
of such addresses. 


4.1.2 Paging-Mode Enabling 


If CRO.PG = 1, a logical processor is in one of four paging modes, depending on the values of CR4.PAE, 
IA32_EFER.LME, and CR4.LA57. Figure 4-1 illustrates how software can enable these modes and make transitions 
between them. The following items identify certain limitations and other details: 


¢ JA32_EFER.LME cannot be modified while paging is enabled (CRO.PG = 1). Attempts to do so using WRMSR 
cause a general-protection exception (#GP(0)). 


¢ Paging cannot be enabled (by setting CRO.PG to 1) while CR4.PAE = 0 and IA32_EFER.LME = 1. Attempts to do 
so using MOV to CRO cause a general-protection exception (#GP(0)). 


* One node in Figure 4-1 is labeled “IA-32e mode.” This node represents either 4-level paging (if CR4.LA57 = 0) 
or 5-level paging (if CR4.LA57 = 1). As noted in the following items, software cannot modify CR4.LA57 
(effecting transition between 4-level paging and 5-level paging) without first disabling paging. 


® CR4.PAE and CR4.LA57 cannot be modified while either 4-level paging or 5-level paging is in use (when 
CRO.PG = 1 and IA32_EFER.LME = 1). Attempts to do so using MOV to CR4 cause a general-protection 
exception (#GP(0)). 


* Regardless of the current paging mode, software can disable paging by clearing CRO.PG with MOV to CRO.! 


¢ Software can transition between 32-bit paging and PAE paging by changing the value of CR4.PAE with MOV to 
CR4. 


° Software cannot transition directly between 4-level paging (or 5-level paging) and any of other paging mode. 
It must first disable paging (by clearing CRO.PG with MOV to CRO), then set CR4.PAE, IA32_EFER.LME, and 
CR4.LA57 to the desired values (with MOV to CR4 and WRMSR), and then re-enable paging (by setting CRO.PG 
with MOV to CRO). As noted earlier, an attempt to modify CR4.PAE, IA32_EFER.LME, or CR.LA57 while 4-level 
paging or 5-level paging is enabled causes a general-protection exception (#GP(0)). 


¢ VMxX transitions allow transitions between paging modes that are not possible using MOV to CR or WRMSR. This 
is because VMX transitions can load CRO, CR4, and IA32_EFER in one operation. See Section 4.11.1. 


1. If the logical processor is in 64-bit mode or if CR4.PCIDE = 1, an attempt to clear CRO.PG causes a general-protection exception 
(#GP). Software should transition to compatibility mode and clear CR4.PCIDE before attempting to disable paging. 
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Figure 4-1. Enabling and Changing Paging Modes 


4.1.3 Paging-Mode Modifiers 


Details of how each paging mode operates are determined by the following control bits: 


The WP flag in CRO (bit 16). 


e 
bit 23, and bit 24, respectively). 
The NXE flag in the IA32_EFER MSR 


(bit 11). 


The PSE, PGE, PCIDE, SMEP, SMAP, PKE, CET, and PKS flags in CR4 (bit 4, bit 7, bit 17, bit 20, bit 21, bit 22, 


CRO.WP allows pages to be protected from supervisor-mode writes. If CRO.WP = 0, supervisor-mode write 
accesses are allowed to linear addresses with read-only access rights; if CRO.WP = 1, they are not. (User-mode 
write accesses are never allowed to linear addresses with read-only access rights, regardless of the value of 


CRO.WP.) Section 4.6 explains how access rights are determined, including the definition of supervisor-mode and 
user-mode accesses. 

CR4.PSE enables 4-MByte pages for 32-bit paging. If CR4.PSE = 0, 32-bit paging can use only 4-KByte pages; if 

CR4.PSE = 1, 32-bit paging can use both 4-KByte pages and 4-MByte pages. See Section 4.3 for more information. 
(PAE paging, 4-level paging, and 5-level paging can use multiple page sizes regardless of the value of CR4.PSE.) 


CR4.PGE enables global pages. If CR4.PGE = 0, no translations are shared across address spaces; if CR4.PGE = 1, 
specified translations may be shared across address spaces. See Section 4.10.2.4 for more information. 


CR4.PCIDE enables process-context identifiers (PCIDs) for 4-level paging and 5-level paging. PCIDs allow a logical 
processor to cache information for multiple linear-address spaces. See Section 4.10.1 for more information. 
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CR4.SMEP allows pages to be protected from supervisor-mode instruction fetches. If CR4.SMEP = 1, software 
operating in supervisor mode cannot fetch instructions from linear addresses that are accessible in user mode. 
Section 4.6 explains how access rights are determined, including the definition of supervisor-mode accesses and 
user-mode accessibility. 


CR4.SMAP allows pages to be protected from supervisor-mode data accesses. If CR4.SMAP = 1, software oper- 
ating in supervisor mode cannot access data at linear addresses that are accessible in user mode. Software can 
override this protection by setting EFLAGS.AC. Section 4.6 explains how access rights are determined, including 
the definition of supervisor-mode accesses and user-mode accessibility. 


CR4.PKE and CR4.PKS enable specification of access rights based on protection keys. 4-level paging and 5-level 
paging associate each linear address with a protection key. When CR4.PKE = 1, the PKRU register specifies, for 
each protection key, whether user-mode linear addresses with that protection key can be read or written. When 
CR4.PKS = 1, the IA32_PKRS MSR does the same for supervisor-mode linear addresses. See Section 4.6 for more 
information. 


CR4.CET enables control-flow enforcement technology, including the shadow-stack feature. If CR4.CET = 1, 
certain memory accesses are identified as shadow-stack accesses and certain linear addresses translate to 
shadow-stack pages. Section 4.6 explains how access rights are determined for these accesses and pages. (The 
processor allows CR4.CET to be set only if CRO.WP is also set.) 


IA32_EFER.NXE enables execute-disable access rights for PAE paging, 4-level paging, and 5-level paging. If 
IA32_EFER.NXE = 1, instruction fetches can be prevented from specified linear addresses (even if data reads from 
the addresses are allowed). Section 4.6 explains how access rights are determined. (IA32_EFER.NXE has no effect 
with 32-bit paging. Software that wants to use this feature to limit instruction fetches from readable pages must 
use PAE paging, 4-level paging, or 5-level paging.) 


4.1.4 Enumeration of Paging Features by CPUID 


Software can discover support for different paging features using the CPUID instruction: 


¢ PSE: page-size extensions for 32-bit paging. 
If CPUID.01H:EDX.PSE [bit 3] = 1, CR4.PSE may be set to 1, enabling support for 4-MByte pages with 32-bit 
paging (see Section 4.3). 


¢ PAE: physical-address extension. 
If CPUID.O1H:EDX.PAE [bit 6] = 1, CR4.PAE may be set to 1, enabling PAE paging (this setting is also required 
for 4-level paging and 5-level paging). 


¢ PGE: global-page support. 
If CPUID.01H:EDX.PGE [bit 13] = 1, CR4.PGE may be set to 1, enabling the global-page feature (see Section 
4.10.2.4). 


¢ PAT: page-attribute table. 
If CPUID.01H:EDX.PAT [bit 16] = 1, the 8-entry page-attribute table (PAT) is supported. When the PAT is 
supported, three bits in certain paging-structure entries select a memory type (used to determine type of 
caching used) from the PAT (see Section 4.9.2). 

° =PSE-36: page-size extensions with 40-bit physical-address extension. 
If CPUID.01H:EDX.PSE-36 [bit 17] = 1, the PSE-36 mechanism is supported, indicating that translations using 
4-MByte pages with 32-bit paging may produce physical addresses with up to 40 bits (see Section 4.3). 

¢ PCID: process-context identifiers. 
If CPUID.O1H:ECX.PCID [bit 17] = 1, CR4.PCIDE may be set to 1, enabling process-context identifiers (see 
Section 4.10.1). 

° SMEP: supervisor-mode execution prevention. 
If CPUID.(EAX=07H,ECX=0H):EBX.SMEP [bit 7] = 1, CR4.SMEP may be set to 1, enabling supervisor-mode 
execution prevention (see Section 4.6). 

¢ SMAP: supervisor-mode access prevention. 
If CPUID.(EAX=07H,ECX=0H):EBX.SMAP [bit 20] = 1, CR4.SMAP may be set to 1, enabling supervisor-mode 
access prevention (see Section 4.6). 
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° PKU: protection keys for user-mode pages. 
If CPUID.(EAX=07H,ECX=0H):ECX.PKU [bit 3] = 1, CR4.PKE may be set to 1, enabling protection keys for 
user-mode pages (see Section 4.6). 


° OSPKE: enabling of protection keys for user-mode pages. 
CPUID.(EAX=07H,ECX=0H):ECX.OSPKE [bit 4] returns the value of CR4.PKE. Thus, protection keys for user- 
mode pages are enabled if this flag is 1 (see Section 4.6). 


* CET: control-flow enforcement technology. 
If CPUID.(EAX=07H,ECX=0H):ECX.CET_SS [bit 7] = 1, CR4.CET may be set to 1, enabling shadow-stack 
pages (see Section 4.6). 


° ~=LA57: 57-bit linear addresses and 5-level paging. 
If CPUID.(EAX=07H,ECX=0):ECX.LA57 [bit 16] = 1, CR4.LA57 may be set to 1, enabling 5-level paging. 


¢ PKS: protection keys for supervisor-mode pages. 
If CPUID.(EAX=07H,ECX=0H):ECX.PKS [bit 31] = 1, CR4.PKS may be set to 1, enabling protection keys for 
supervisor-mode pages (see Section 4.6). 


° NX: execute disable. 
If CPUID.80000001H:EDX.NX [bit 20] = 1, IA32_EFER.NXE may be set to 1, allowing software to disable 
execute access to selected pages (see Section 4.6). (Processors that do not support CPUID function 
80000001H do not allow IA32_EFER.NXE to be set to 1.) 

° Page1iGB: 1-GByte pages. 
If CPUID.80000001H:EDX.Page1GB [bit 26] = 1, 1-GByte pages may be supported with 4-level paging and 5- 
level paging (see Section 4.5). 

¢ LM: IA-32e mode support. 
If CPUID.80000001H:EDX.LM [bit 29] = 1, IA32_EFER.LME may be set to 1, enabling IA-32e mode (with either 
4-level paging or 5-level paging). (Processors that do not support CPUID function 80000001H do not allow 
IA32_EFER.LME to be set to 1.) 

¢ CPUID.80000008H:EAX[7:0] reports the physical-address width supported by the processor. (For processors 
that do not support CPUID function 80000008H, the width is generally 36 if CPUID.O1H:EDX.PAE [bit 6] = 1 
and 32 otherwise.) This width is referred to as MAXPHYADDR. MAXPHYADDR is at most 52. 

¢ CPUID.80000008H:EAX[15:8] reports the linear-address width supported by the processor. Generally, this 
value is reported as follows: 


— If CPUID.80000001H:EDX.LM [bit 29] = 0, the value is reported as 32. 


— If CPUID.80000001H:EDX.LM [bit 29] = 1 and CPUID.(EAX=07H,ECX=0):ECX.LA57 [bit 16] = 0, the 
value is reported as 48. 


— If CPUID.(EAX=07H,ECX=0):ECX.LA57 [bit 16] = 1, the value is reported as 57. 
(Processors that do not support CPUID function 80000008H, support a linear-address width of 32.) 


4.2 HIERARCHICAL PAGING STRUCTURES: AN OVERVIEW 


All four paging modes translate linear addresses using hierarchical paging structures. This section provides an 
overview of their operation. Section 4.3, Section 4.4, Section 4.5, and Section 4.6 provide details for the four 
paging modes. 


Every paging structure is 4096 Bytes in size and comprises a number of individual entries. With 32-bit paging, 
each entry is 32 bits (4 bytes); there are thus 1024 entries in each structure. With the other paging modes, each 
entry is 64 bits (8 bytes); there are thus 512 entries in each structure. (PAE paging includes one exception, a 
paging structure that is 32 bytes in size, containing 4 64-bit entries. ) 


The processor uses the upper portion of a linear address to identify a series of paging-structure entries. The last of 
these entries identifies the physical address of the region to which the linear address translates (called the page 
frame). The lower portion of the linear address (called the page offset) identifies the specific address within that 
region to which the linear address translates. 
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Each paging-structure entry contains a physical address, which is either the address of another paging structure or 
the address of a page frame. In the first case, the entry is said to reference the other paging structure; in the 
latter, the entry is said to map a page. 


The first paging structure used for any translation is located at the physical address in CR3. A linear address is 
translated using the following iterative procedure. A portion of the linear address (initially the uppermost bits) 
selects an entry in a paging structure (initially the one located using CR3). If that entry references another paging 
structure, the process continues with that paging structure and with the portion of the linear address immediately 
below that just used. If instead the entry maps a page, the process completes: the physical address in the entry is 
that of the page frame and the remaining lower portion of the linear address is the page offset. 


The following items give an example for each of the four paging modes (each example locates a 4-KByte page 
frame): 


* With 32-bit paging, each paging structure comprises 1024 = 2!° entries. For this reason, the translation 
process uses 10 bits at a time from a 32-bit linear address. Bits 31:22 identify the first paging-structure entry 
and bits 21:12 identify a second. The latter identifies the page frame. Bits 11:0 of the linear address are the 
page offset within the 4-KByte page frame. (See Figure 4-2 for an illustration.) 


* With PAE paging, the first paging structure comprises only 4 = 2? entries. Translation thus begins by using 
bits 31:30 from a 32-bit linear address to identify the first paging-structure entry. Other paging structures 
comprise 512 =2° entries, so the process continues by using 9 bits at a time. Bits 29:21 identify a second 
paging-structure entry and bits 20:12 identify a third. This last identifies the page frame. (See Figure 4-5 for 
an illustration. ) 


* With 4-level paging, each paging structure comprises 512 = 2° entries and translation uses 9 bits at a time 
from a 48-bit linear address. Bits 47:39 identify the first paging-structure entry, bits 38:30 identify a second, 
bits 29:21 a third, and bits 20:12 identify a fourth. Again, the last identifies the page frame. (See Figure 4-8 
for an illustration.) 


e =5-level paging is similar to 4-level paging except that 5-level paging translates 57-bit linear addresses. 
Bits 56:48 identify the first paging-structure entry, while the remaining bits are used as with 4-level paging. 


The translation process in each of the examples above completes by identifying a page frame; the page frame is 
part of the translation of the original linear address. In some cases, however, the paging structures may be 
configured so that the translation process terminates before identifying a page frame. This occurs if the process 
encounters a paging-structure entry that is marked “not present” (because its P flag — bit O — is clear) or in which 
a reserved bit is set. In this case, there is no translation for the linear address; an access to that address causes a 
page-fault exception (see Section 4.7). 


In the examples above, a paging-structure entry maps a page with a 4-KByte page frame when only 12 bits remain 
in the linear address; entries identified earlier always reference other paging structures. That may not apply in 
other cases. The following items identify when an entry maps a page and when it references another paging struc- 
ture: 


¢ If more than 12 bits remain in the linear address, bit 7 (PS — page size) of the current paging-structure entry 
is consulted. If the bit is 0, the entry references another paging structure; if the bit is 1, the entry maps a page. 


¢ If only 12 bits remain in the linear address, the current paging-structure entry always maps a page (bit 7 is 
used for other purposes). 


If a paging-structure entry maps a page when more than 12 bits remain in the linear address, the entry identifies 
a page frame larger than 4 KBytes. For example, 32-bit paging uses the upper 10 bits of a linear address to locate 
the first paging-structure entry; 22 bits remain. If that entry maps a page, the page frame is 272 Bytes = 4 MBytes. 
32-bit paging can use 4-MByte pages if CR4.PSE = 1. The other paging modes can use 2-MByte pages (regardless 
of the value of CR4.PSE). 4-level paging and 5-level paging can use 1-GByte pages if the processor supports them 
(see Section 4.1.4). 


Paging structures are given different names based on their uses in the translation process. Table 4-2 gives the 
names of the different paging structures. It also provides, for each structure, the source of the physical address 
used to locate it (CR3 or a different paging-structure entry); the bits in the linear address used to select an entry 
from the structure; and details of whether and how such an entry can map a page. 
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Table 4-2. Paging Structures in the Different Paging Modes 


Physical 
Paging Structure Paging Mode Address of 
Structure 


Bits Selecting 


Entry Page Mapping 


32-bit, PAE, 4-level N/A 


PML5 table 


5-level N/A (PS must be 0) 
32-bit, PAE 


PML4 table 4-level 


N/A (PS must be 0) 
5-level 


32-bit N/A 


Page-directory- PAE N/A (PS must be 0) 
pointer table 


4-level, 5-level 1-GByte page if PS=1 ! 


4-MByte page if PS=1¢ 


Page directory 
PAE, 4-level, 5-level 2-MByte page if PS=1 


32-bit 
Page table 4-KByte page 
PAE, 4-level, 5-level 


NOTES: 

1. Not all processors support 1-GByte pages; see Section 4.1.4. 

2. 32-bit paging ignores the PS flag in a PDE (and uses the entry to reference a page table) unless CR4.PSE = 1. Not all processors sup- 
port 4-MByte pages with 32-bit paging; see Section 4.1.4. 


4.3 32-BIT PAGING 


A logical processor uses 32-bit paging if CRO.PG = 1 and CR4.PAE = 0. 32-bit paging translates 32-bit linear 
addresses to 40-bit physical addresses.! Although 40 bits corresponds to 1 TByte, linear addresses are limited to 
32 bits; at most 4 GBytes of linear-address space may be accessed at any given time. 


32-bit paging uses a hierarchy of paging structures to produce a translation for a linear address. CR3 is used to 
locate the first paging-structure, the page directory. Table 4-3 illustrates how CR3 is used with 32-bit paging. 


32-bit paging may map linear addresses to either 4-KByte pages or 4-MByte pages. Figure 4-2 illustrates the trans- 
lation process when it uses a 4-KByte page; Figure 4-3 covers the case of a 4-MByte page. The following items 
describe the 32-bit paging process in more detail as well has how the page size is determined: 


¢ A4-KByte naturally aligned page directory is located at the physical address specified in bits 31:12 of CR3 (see 
Table 4-3). A page directory comprises 1024 32-bit entries (PDEs). A PDE is selected using the physical address 
defined as follows: 


— Bits 39:32 are all 0. 
— Bits 31:12 are from CR3. 
— Bits 11:2 are bits 31:22 of the linear address. 


1. Bits in the range 39:32 are O in any physical address used by 32-bit paging except those used to map 4-MByte pages. If the proces- 
sor does not support the PSE-36 mechanism, this is true also for physical addresses used to map 4-MByte pages. If the processor 
does support the PSE-36 mechanism and MAXPHYADDR < 40, bits in the range 39:MAXPHYADDR are 0 in any physical address used 
to map a 4-MByte page. (The corresponding bits are reserved in PDEs.) See Section 4.1.4 for how to determine MAXPHYADDR and 
whether the PSE-36 mechanism is supported. 
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— Bits 1:0 are O. 


Because a PDE is identified using bits 31:22 of the linear address, it controls access to a 4-Mbyte region of the 
linear-address space. Use of the PDE depends on CR4.PSE and the PDE’s PS flag (bit 7): 


°¢ If CR4.PSE = 1 and the PDE’s PS flag is 1, the PDE maps a 4-MByte page (see Table 4-4). The final physical 
address is computed as follows: 


— Bits 39:32 are bits 20:13 of the PDE. 
— Bits 31:22 are bits 31:22 of the PDE. 
— Bits 21:0 are from the original linear address. 


¢ If CR4.PSE = 0 or the PDE's PS flag is 0, a 4-KByte naturally aligned page table is located at the physical 
address specified in bits 31:12 of the PDE (see Table 4-5). A page table comprises 1024 32-bit entries (PTEs). 
A PTE is selected using the physical address defined as follows: 


— Bits 39:32 are all 0. 

— Bits 31:12 are from the PDE. 

— Bits 11:2 are bits 21:12 of the linear address. 
— Bits 1:0 are O. 


° Because a PTE is identified using bits 31:12 of the linear address, every PTE maps a 4-KByte page (see 
Table 4-6). The final physical address is computed as follows: 


— Bits 39:32 are all 0. 
— Bits 31:12 are from the PTE. 
— Bits 11:0 are from the original linear address. 


If a paging-structure entry’s P flag (bit 0) is O or if the entry sets any reserved bit, the entry is used neither to refer- 
ence another paging-structure entry nor to map a page. There is no translation for a linear address whose transla- 
tion would use such a paging-structure entry; a reference to such a linear address causes a page-fault exception 
(see Section 4.7). 


With 32-bit paging, there are reserved bits only if CR4.PSE = 1: 


¢ Ifthe P flag and the PS flag (bit 7) of a PDE are both 1, the bits reserved depend on MAXPHYADDR, and whether 
the PSE-36 mechanism is supported: 


— Ifthe PSE-36 mechanism is not supported, bits 21:13 are reserved. 


— If the PSE-36 mechanism is supported, bits 21:(M-19) are reserved, where M is the minimum of 40 and 
MAXPHYADDR. 


* If the PAT is not supported:? 

— Ifthe P flag of a PTE is 1, bit 7 is reserved. 

— Ifthe P flag and the PS flag of a PDE are both 1, bit 12 is reserved. 
(If CR4.PSE = 0, no bits are reserved with 32-bit paging.) 


A reference using a linear address that is successfully translated to a physical address is performed only if allowed 
by the access rights of the translation; see Section 4.6. 


1. The upper bits in the final physical address do not all come from corresponding positions in the PDE; the physical-address bits in the 
PDE are not all contiguous. 


2. See Section 4.1.4 for how to determine MAXPHYADDR and whether the PSE-36 mechanism is supported. 
3. See Section 4.1.4 for how to determine whether the PAT is supported. 
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Figure 4-2. Linear-Address Translation to a 4-KByte Page using 32-Bit Paging 
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Figure 4-3. Linear-Address Translation to a 4-MByte Page using 32-Bit Paging 


Figure 4-4 gives a summary of the formats of CR3 and the paging-structure entries with 32-bit paging. For the 
paging structure entries, it identifies separately the format of entries that map pages, those that reference other 
paging structures, and those that do neither because they are “not present”; bit 0 (P) and bit 7 (PS) are highlighted 
because they determine how such an entry is used. 
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Figure 4-4. Formats of CR3 and Paging-Structure Entries with 32-Bit Paging 


1. CR3 has 64 bits on processors supporting the Intel-64 architecture. These bits are ignored with 32-bit paging. 
2. This example illustrates a processor in which MAXPHYADDR is 36. If this value is larger or smaller, the number of bits reserved in 
positions 20:13 of a PDE mapping a 4-MByte page will change. 


Table 4-3. Use of CR3 with 32-Bit Paging 


Bit Contents 

Position(s) 

2:0 Ignored 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page directory during linear- 
address translation (See Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page directory during linear- 
address translation (See Section 4.9) 

11:5 Ignored 

31:12 Physical address of the 4-KByte aligned page directory used for linear-address translation 

63:32 Ignored (these bits exist only on processors supporting the Intel-64 architecture) 
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Table 4-4. Format of a 32-Bit Page-Directory Entry that Maps a 4-MByte Page 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 4-MByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 4-MByte page referenced by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 4-MByte page referenced by this entry (see Section 
4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 4-MByte page referenced by 
this entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 4-MByte page referenced by 
this entry (see Section 4.9) 

5 (A) Accessed; indicates whether software has accessed the 4-MByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 4-MByte page referenced by this entry (see Section 4.8) 

7 (PS) Page size; must be 1 (otherwise, this entry references a page table; see Table 4-5) 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 

11:9 Ignored 

12 (PAT) If the PAT is supported, indirectly determines the memory type used to access the 4-MByte page referenced by this 
entry (see Section 4.9.2); otherwise, reserved (must be 0)! 

(M-20):13 Bits (M-1):32 of physical address of the 4-MByte page referenced by this entry? 

21:(M-19) Reserved (must be 0) 

31:22 Bits 31:22 of physical address of the 4-MByte page referenced by this entry 

NOTES: 


1. See Section 4.1.4 for how to determine whether the PAT is supported. 


2. If the PSE-36 mechanism is not supported, M is 32, and this row does not apply. If the PSE-36 mechanism is supported, M is the min- 
imum of 40 and MAXPHYADDR (this row does not apply if MAXPHYADDR = 32). See Section 4.1.4 for how to determine MAXPHYA- 
DDR and whether the PSE-36 mechanism is supported. 
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Table 4-5. Format of a 32-Bit Page-Directory Entry that References a Page Table 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a page table 

1 (R/W) Read/write; if 0, writes may not be allowed to the 4-MByte region controlled by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 4-MByte region controlled by this entry (see Section 
4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page table referenced by this 
entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page table referenced by this 
entry (see Section 4.9) 

5 (A) Accessed; indicates whether this entry has been used for linear-address translation (see Section 4.8) 

6 Ignored 

7 (PS) If CR4.PSE = 1, must be O (otherwise, this entry maps a 4-MByte page; see Table 4-4); otherwise, ignored 

11:8 Ignored 

31:12 Physical address of 4-KByte aligned page table referenced by this entry 

Table 4-6. Format of a 32-Bit Page-Table Entry that Maps a 4-KByte Page 

Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 4-KByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 4-KByte page referenced by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 4-KByte page referenced by this entry (see Section 
4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 4-KByte page referenced by this 
entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 4-KByte page referenced by this 
entry (see Section 4.9) 

5 (A) Accessed; indicates whether software has accessed the 4-KByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 4-KByte page referenced by this entry (see Section 4.8) 

7 (PAT) If the PAT is supported, indirectly determines the memory type used to access the 4-KByte page referenced by this 
entry (see Section 4.9.2); otherwise, reserved (must be 0)! 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 

11:9 Ignored 

31:12 Physical address of the 4-KByte page referenced by this entry 

NOTES: 


1. See Section 4.1.4 for how to determine whether the PAT is supported. 


Vol.3A 4-13 


PAGING 


4.4 PAE PAGING 


A logical processor uses PAE paging if CRO.PG = 1, CR4.PAE = 1, and IA32_EFER.LME = 0. PAE paging translates 
32-bit linear addresses to 52-bit physical addresses.! Although 52 bits corresponds to 4 PBytes, linear addresses 
are limited to 32 bits; at most 4 GBytes of linear-address space may be accessed at any given time. 


With PAE paging, a logical processor maintains a set of four (4) PDPTE registers, which are loaded from an address 
in CR3. Linear address are translated using 4 hierarchies of in-memory paging structures, each located using one 
of the PDPTE registers. (This is different from the other paging modes, in which there is one hierarchy referenced 
by CR3.) 


Section 4.4.1 discusses the PDPTE registers. Section 4.4.2 describes linear-address translation with PAE paging. 


4.4.1 PDPTE Registers 


When PAE paging is used, CR3 references the base of a 32-Byte page-directory-pointer table. Table 4-7 illus- 
trates how CR3 is used with PAE paging. 


Table 4-7. Use of CR3 with PAE Paging 


Bit Contents 

Position(s) 

4:0 Ignored 

31:5 Physical address of the 32-Byte aligned page-directory-pointer table used for linear-address translation 
63:32 Ignored (these bits exist only on processors supporting the Intel-64 architecture) 


The page-directory-pointer-table comprises four (4) 64-bit entries called PDPTEs. Each PDPTE controls access to a 
1-GByte region of the linear-address space. Corresponding to the PDPTEs, the logical processor maintains a set of 
four (4) internal, non-architectural PDPTE registers, called PDPTEO, PDPTE1, PDPTE2, and PDPTE3. The logical 
processor loads these registers from the PDPTEs in memory as part of certain operations: 


¢ If PAE paging would be in use following an execution of MOV to CRO or MOV to CR4 (see Section 4.1.1) and the 
instruction is modifying any of CRO.CD, CRO.NW, CRO.PG, CR4.PAE, CR4.PGE, CR4.PSE, or CR4.SMEP; then the 
PDPTEs are loaded from the address in CR3. 


¢ If MOV to CR3 is executed while the logical processor is using PAE paging, the PDPTEs are loaded from the 
address being loaded into CR3. 


¢ If PAE paging is in use and a task switch changes the value of CR3, the PDPTEs are loaded from the address in 
the new CR3 value. 


* Certain VMX transitions load the PDPTE registers. See Section 4.11.1. 
Table 4-8 gives the format of a PDPTE. If any of the PDPTEs sets both the P flag (bit 0) and any reserved bit, the 


MOV to CR instruction causes a general-protection exception (#GP(0)) and the PDPTEs are not loaded.? As shown 
in Table 4-8, bits 2:1, 8:5, and 63:MAXPHYADDR are reserved in the PDPTEs. 


1. If MAXPHYADDR < 52, bits in the range 51:MAXPHYADDR will be 0 in any physical address used by PAE paging. (The corresponding 
bits are reserved in the paging-structure entries.) See Section 4.1.4 for how to determine MAXPHYADDR. 


2. Onsome processors, reserved bits are checked even in PDPTEs in which the P flag (bit 0) is 0. 
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Table 4-8. Format of a PAE Page-Directory-Pointer-Table Entry (PDPTE) 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a page directory 

2:1 Reserved (must be 0) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page directory referenced by 
this entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page directory referenced by 
this entry (see Section 4.9) 

8:5 Reserved (must be 0) 

11:9 Ignored 


(M-1):12 Physical address of 4-KByte aligned page directory referenced by this entry! 


63:M Reserved (must be 0) 


NOTES: 
1. Mis an abbreviation for MAXPHYADDR, which is at most 52; see Section 4.1.4. 


4.4.2 Linear-Address Translation with PAE Paging 


PAE paging may map linear addresses to either 4-KByte pages or 2-MByte pages. Figure 4-5 illustrates the trans- 
lation process when it produces a 4-KByte page; Figure 4-6 covers the case of a 2-MByte page. The following items 
describe the PAE paging process in more detail as well has how the page size is determined: 


¢ Bits 31:30 of the linear address select a PDPTE register (see Section 4.4.1); this is PDPTE/, where / is the value 
of bits 31:30.! Because a PDPTE register is identified using bits 31:30 of the linear address, it controls access 
to a 1-GByte region of the linear-address space. If the P flag (bit 0) of PDPTE/ is 0, the processor ignores bits 
63:1, and there is no mapping for the 1-GByte region controlled by PDPTE/. A reference using a linear address 
in this region causes a page-fault exception (see Section 4.7). 


° If the P flag of PDPTE/ is 1, 4-KByte naturally aligned page directory is located at the physical address specified 
in bits 51:12 of PDPTE/ (see Table 4-8 in Section 4.4.1). A page directory comprises 512 64-bit entries (PDEs). 
A PDE is selected using the physical address defined as follows: 


— Bits 51:12 are from PDPTE/. 
— Bits 11:3 are bits 29:21 of the linear address. 
— Bits 2:0 are O. 


Because a PDE is identified using bits 31:21 of the linear address, it controls access to a 2-Mbyte region of the 
linear-address space. Use of the PDE depends on its PS flag (bit 7): 


¢ If the PDE’s PS flag is 1, the PDE maps a 2-MByte page (see Table 4-9). The final physical address is computed 
as follows: 


— Bits 51:21 are from the PDE. 
— Bits 20:0 are from the original linear address. 


e If the PDE’s PS flag is 0, a 4-KByte naturally aligned page table is located at the physical address specified in 
bits 51:12 of the PDE (see Table 4-10). A page table comprises 512 64-bit entries (PTEs). A PTE is selected 
using the physical address defined as follows: 


— Bits 51:12 are from the PDE. 


1. With PAE paging, the processor does not use CR3 when translating a linear address (as it does in the other paging modes). It does 
not access the PDPTEs in the page-directory-pointer table during linear-address translation. 
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— Bits 11:3 are bits 20:12 of the linear address. 
— Bits 2:0 are O. 


° Because a PTE is identified using bits 31:12 of the linear address, every PTE maps a 4-KByte page (see 
Table 4-11). The final physical address is computed as follows: 


— Bits 51:12 are from the PTE. 
— Bits 11:0 are from the original linear address. 


If the P flag (bit 0) of a PDE or a PTE is 0 or if a PDE or a PTE Sets any reserved bit, the entry is used neither to 
reference another paging-structure entry nor to map a page. There is no translation for a linear address whose 
translation would use such a paging-structure entry; a reference to such a linear address causes a page-fault 
exception (see Section 4.7). 


The following bits are reserved with PAE paging: 
° If the P flag (bit 0) of a PDE or a PTE is 1, bits 62: MAXPHYADDR are reserved. 
e If the P flag and the PS flag (bit 7) of a PDE are both 1, bits 20:13 are reserved. 
° If IA32_EFER.NXE = 0 and the P flag of a PDE or a PTE is 1, the XD flag (bit 63) is reserved. 
* If the PAT is not supported:! 
— Ifthe P flag of a PTE is 1, bit 7 is reserved. 
— Ifthe P flag and the PS flag of a PDE are both 1, bit 12 is reserved. 


A reference using a linear address that is successfully translated to a physical address is performed only if allowed 
by the access rights of the translation; see Section 4.6. 


Linear Address 
31 30 29 21 20 12 11 0 


Directory Table Offset 


Directory Pointer —» 


12 4-KByte Page 


Page Table Physical Address 
Page Directory 9 
9 \_y | PTE ri 
|_| PDE with PS=0 ie ae 
2 


PDPTE Registers 


40 


——* | PDPTE value 


Figure 4-5. Linear-Address Translation to a 4-KByte Page using PAE Paging 


1. See Section 4.1.4 for how to determine whether the PAT is supported. 
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Linear Address 
31 30 29 21 20 0 
Directory Offset 


Directory 
Pointer 


21  2-MByte Page 


Page Directory —>) Physical Address 


PDPTE Registers 


—~| PDE with PS=1 


— | PDPTE value > 
40 


31 


Figure 4-6. Linear-Address Translation to a 2-MByte Page using PAE Paging 


Table 4-9. Format of a PAE Page-Directory Entry that Maps a 2-MByte Page 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 2-MByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 2-MByte page referenced by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 2-MByte page referenced by this entry (see Section 
46) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 2-MByte page referenced by 
this entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 2-MByte page referenced by this 
entry (see Section 4.9) 

5 (A) Accessed; indicates whether software has accessed the 2-MByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 2-MByte page referenced by this entry (see Section 4.8) 

7 (PS) Page size; must be 1 (otherwise, this entry references a page table; see Table 4-10) 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 

11:9 Ignored 

12 (PAT) If the PAT is supported, indirectly determines the memory type used to access the 2-MByte page referenced by this 
entry (see Section 4.9.2); otherwise, reserved (must be 0)! 

20:13 Reserved (must be 0) 

(M-1):21 Physical address of the 2-MByte page referenced by this entry 

62:M Reserved (must be 0) 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 2-MByte page controlled by 
this entry; see Section 4.6); otherwise, reserved (must be 0) 

NOTES: 


1. See Section 4.1.4 for how to determine whether the PAT is supported. 
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Table 4-10. Format of a PAE Page-Directory Entry that References a Page Table 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a page table 

1 (R/W) Read/write; if 0, writes may not be allowed to the 2-MByte region controlled by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 2-MByte region controlled by this entry (see 
Section 4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page table referenced by this 
entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page table referenced by this 
entry (see Section 4.9) 

5 (A) Accessed; indicates whether this entry has been used for linear-address translation (see Section 4.8) 

6 Ignored 

7 (PS) Page size; must be O (otherwise, this entry maps a 2-MByte page; see Table 4-9) 

11:8 Ignored 

(M-1):12 Physical address of 4-KByte aligned page table referenced by this entry 

62:M Reserved (must be 0) 

63 (XD) If [A32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 2-MByte region controlled 
by this entry; see Section 4.6); otherwise, reserved (must be 0) 

Table 4-11. Format of a PAE Page-Table Entry that Maps a 4-KByte Page 

Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 4-KByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 4-KByte page referenced by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 4-KByte page referenced by this entry (see Section 
4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 4-KByte page referenced by 
this entry (see Section 4.9) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 4-KByte page referenced by this 
entry (see Section 4.9) 

5 (A) Accessed; indicates whether software has accessed the 4-KByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 4-KByte page referenced by this entry (see Section 4.8) 

7 (PAT) If the PAT is supported, indirectly determines the memory type used to access the 4-KByte page referenced by this 
entry (see Section 4.9.2); otherwise, reserved (must be 0)! 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 
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Table 4-11. Format of a PAE Page-Table Entry that Maps a 4-KByte Page (Contd.) 


Bit Contents 
Position(s) 
11:9 Ignored 


(M-1):12 Physical address of the 4-KByte page referenced by this entry 


62:M Reserved (must be 0) 


63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 4-KByte page controlled by 
this entry; see Section 4.6); otherwise, reserved (must be 0) 


NOTES: 
1. See Section 4.1.4 for how to determine whether the PAT is supported. 


Figure 4-7 gives a summary of the formats of CR3 and the paging-structure entries with PAE paging. For the paging 
structure entries, it identifies separately the format of entries that map pages, those that reference other paging 
structures, and those that do neither because they are “not present”; bit O (P) and bit 7 (PS) are highlighted 
because they determine how a paging-structure entry is used. 


serogazesaszi| || | brosezesssziiosacsaaeiolbleseazilol 


Ignored@ Address of page-directory-pointer table Ignored 


3 PDPTE: 
Reserved Address of page directory . present 


PDTPE: 


Ignored not 
present 


PDE: 


Address of 
Reserved 2MB page frame Reserved ‘ : ae 


PDE: 


Reserved Address of page table : page 
table 


PDE: 


Ignored not 
present 


PTE: 
Reserved Address of 4KB page frame : 4KB 
page 
PTE: 


Ignored not 
present 


Figure 4-7. Formats of CR3 and Paging-Structure Entries with PAE Paging 


NOTES: 

1. Mis an abbreviation for MAXPHYADDR. 

2. CR3 has 64 bits only on processors supporting the Intel-64 architecture. These bits are ignored with PAE paging. 
3. Reserved fields must be 0. 

4. If IA32_EFER.NXE = 0 and the P flag of a PDE or a PTE is 1, the XD flag (bit 63) is reserved. 
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4.5 4-LEVEL PAGING AND 5-LEVEL PAGING 


Because the operation of 4-level paging and 5-level paging is very similar, they are described together in this 
section. The following items highlight the distinctions between the two paging modes: 


¢ A logical processor uses 4-level paging if CRO.PG = 1, CR4.PAE = 1, IA32_EFER.LME = 1, and CR4.LA57 = 0. 
4-level paging translates 48-bit linear addresses to 52-bit physical addresses.! Although 52 bits corresponds to 
4 PBytes, linear addresses are limited to 48 bits; at most 256 TBytes of linear-address space may be accessed 
at any given time. 


¢ A logical processor uses 5-level paging if CRO.PG = 1, CR4.PAE = 1, IA32_EFER.LME = 1, and CR4.LA57 = 1. 
5-level paging translates 57-bit linear addresses to 52-bit physical addresses. Thus, 5-level paging supports a 
linear-address space sufficient to access the entire physical-address space. 


Both paging modes translate linear addresses using a hierarchy of in-memory paging structures located using the 
contents of CR3, which is used to locate the first paging-structure. For 4-level paging, this is the PML4 table, and 
for 5-level paging it is the PML5 table. Use of CR3 with 4-level paging and 5-level paging depends on whether 
process-context identifiers (PCIDs) have been enabled by setting CR4.PCIDE: 


* Table 4-12 illustrates how CR3 is used with 4-level paging and 5-level paging if CR4.PCIDE = 0. 


Table 4-12. Use of CR3 with 4-Level Paging and 5-level Paging and CR4.PCIDE = 0 


Bit Contents 

Position(s) 

2:0 Ignored 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the PML4 table during linear- 
address translation (see Section 4.9.2) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the PML4 table during linear-address 
translation (See Section 4.9.2) 

11:5 Ignored 

M-1:12 Physical address of the 4-KByte aligned PML4 table or PML5 table used for linear-address translation! 

63:M Reserved (must be 0) 

NOTES: 


1.Mis an abbreviation for MAXPHYADDR, which is at most 52; see Section 4.1.4. 


°* Table 4-13 illustrates how CR3 is used with 4-level paging and 5-level paging if CR4.PCIDE = 1. 


Table 4-13. Use of CR3 with 4-Level Paging and 5-Level Paging and CR4.PCIDE = 1 


Bit Contents 
Position(s) 
11:0 PCID (see Section 4.10.1)! 
M-1:12 Physical address of the 4-KByte aligned PML4 table used for linear-address translation@ 
63:M Reserved (must be 0)3 
NOTES: 


1. Section 4.9.2 explains how the processor determines the memory type used to access the PML4 table during linear-address transla- 
tion with CR4.PCIDE = 1. 


2. Mis an abbreviation for MAXPHYADDR, which is at most 52; see Section 4.1.4. 


1. If MAXPHYADDR < 52, bits in the range 51:MAXPHYADDR will be O in any physical address used by 4-level paging. (The correspond- 
ing bits are reserved in the paging-structure entries.) See Section 4.1.4 for how to determine MAXPHYADDR. 
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3. See Section 4.10.4.1 for use of bit 63 of the source operand of the MOV to CR3 instruction. 


After software modifies the value of CR4.PCIDE, the logical processor immediately begins using CR3 as specified 
for the new value. For example, if software changes CR4.PCIDE from 1 to 0, the current PCID immediately changes 
from CR3[11:0] to OOOH (see also Section 4.10.4.1). In addition, the logical processor subsequently determines 
the memory type used to access the PML4 table using CR3.PWT and CR3.PCD, which had been bits 4:3 of the PCID. 


4-level paging and 5-level paging may map linear addresses to 4-KByte pages, 2-MByte pages, or 1-GByte pages.! 
Figure 4-8 illustrates the translation process for 4-level paging when it produces a 4-KByte page; Figure 4-9 covers 
the case of a 2-MByte page, and Figure 4-10 the case of a 1-GByte page. (The process for 5-level paging is similar. ) 


Linear Address 
47 39 38 30 29 21 20 12 11 0 
PML4 Directory Ptr Directory Table Offset 


9 
79 42_4-KByte Page 
“~»| Physical Addr 


PTE 
Page-Directory- || PDE with PS=0 a0 
Pointer Table 40 Page Table 
Page-Directory 
|_| PDPTE b 40 
9 
40 
>| PML4E — 
> 
40 
CR3 


Figure 4-8. Linear-Address Translation to a 4-KByte Page using 4-Level Paging 


1. Not all processors support 1-GByte pages; see Section 4.1.4. 
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Linear Address 
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Figure 4-9. Linear-Address Translation to a 2-MByte Page using 4-Level Paging 
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Figure 4-10. Linear-Address Translation to a 1-GByte Page using 4-Level Paging 
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4-level paging and 5-level paging associate with each linear address a protection key. Section 4.6 explains how 
the processor uses the protection key in its determination of the access rights of each linear address. 


The remainder of this section describes the translation process used by 4-level paging and 5-level paging in more 
detail, as well has how the page size and protection key are determined. Because the process used by the two 
paging modes is similar, they are described together, with any differences identified, in the following items: 


With 5-level paging, a 4-KByte naturally aligned PMLS5 table is located at the physical address specified in 
bits 51:12 of CR3 (see Table 4-12). (4-level paging does not use a PML5 table and omits this step.) A PML5 
table comprises 512 64-bit entries (PML5Es). A PML5E is selected using the physical address defined as follows: 


— Bits 51:12 are from CR3. 
— Bits 11:3 are bits 56:48 of the linear address. 
— Bits 2:0 are all 0. 


Because a PML5E is identified using bits 56:48 of the linear address, it controls access to a 256-TByte region of 
the linear-address space. 


A 4-KByte naturally aligned PML4 table is located at the physical address specified in bits 51:12 of CR3 (for 4- 
level paging; see Table 4-12) or in bits 51:12 of the PML4E (for 5-level paging; see Table 4-14). A PML4 table 
comprises 512 64-bit entries (PML4Es). A PML4E is selected using the physical address defined as follows: 


— Bits 51:12 are from CR3 (for 4-level paging) or in bits 51:12 of the PML4E (for 5-level paging). 
— Bits 11:3 are bits 47:39 of the linear address. 
— Bits 2:0 are all 0. 


Because a PML4E is identified using bits 47:39 of the linear address, it controls access to a 512-GByte region 
of the linear-address space. 


A 4-KByte naturally aligned page-directory-pointer table is located at the physical address specified in 
bits 51:12 of the PML4E (see Table 4-15). A page-directory-pointer table comprises 512 64-bit entries 
(PDPTEs). A PDPTE is selected using the physical address defined as follows: 


— Bits 51:12 are from the PML4E. 
— Bits 11:3 are bits 38:30 of the linear address. 
— Bits 2:0 are all 0. 


Because a PDPTE is identified using bits 47:30 of the linear address, it controls access to a 1-GByte region of the 
linear-address space. Use of the PDPTE depends on its PS flag (bit 7): 


If the PDPTE’s PS flag is 1, the PDPTE maps a 1-GByte page (see Table 4-16). The final physical address is 
computed as follows: 


— Bits 51:30 are from the PDPTE. 
— Bits 29:0 are from the original linear address. 
The linear address’s protection key is the value of bits 62:59 of the PDPTE (see Section 4.6.2). 


If the PDPTE’s PS flag is 0, a 4-KByte naturally aligned page directory is located at the physical address 
specified in bits 51:12 of the PDPTE (see Table 4-17). A page directory comprises 512 64-bit entries (PDEs). A 
PDE is selected using the physical address defined as follows: 


— Bits 51:12 are from the PDPTE. 
— Bits 11:3 are bits 29:21 of the linear address. 
— Bits 2:0 are all 0. 


Because a PDE is identified using bits 47:21 of the linear address, it controls access to a 2-MByte region of the 
linear-address space. Use of the PDE depends on its PS flag: 


If the PDE's PS flag is 1, the PDE maps a 2-MByte page (see Table 4-18). The final physical address is computed 
as follows: 


1. 


The PS flag of a PDPTE is reserved and must be 0 (if the P flag is 1) if 1-GByte pages are not supported. See Section 4.1.4 for how 
to determine whether 1-GByte pages are supported. 
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— Bits 51:21 are from the PDE. 
— Bits 20:0 are from the original linear address. 
The linear address’s protection key is the value of bits 62:59 of the PDE (see Section 4.6.2). 


¢ If the PDE’s PS flag is 0, a 4-KByte naturally aligned page table is located at the physical address specified in 
bits 51:12 of the PDE (see Table 4-19). A page table comprises 512 64-bit entries (PTEs). A PTE is selected 
using the physical address defined as follows: 


— Bits 51:12 are from the PDE. 
— Bits 11:3 are bits 20:12 of the linear address. 
— Bits 2:0 are all 0. 


° Because a PTE is identified using bits 47:12 of the linear address, every PTE maps a 4-KByte page (see 
Table 4-20). The final physical address is computed as follows: 


— Bits 51:12 are from the PTE. 
— Bits 11:0 are from the original linear address. 
The linear address’s protection key is the value of bits 62:59 of the PTE (see Section 4.6.2). 


If a paging-structure entry’s P flag (bit 0) is O or if the entry sets any reserved bit, the entry is used neither to refer- 
ence another paging-structure entry nor to map a page. There is no translation for a linear address whose transla- 
tion would use such a paging-structure entry; a reference to such a linear address causes a page-fault exception 
(see Section 4.7). 


The following bits in a paging-structure entry are reserved with 4-level paging and 5-level paging (assuming that 
the entry’s P flag is 1): 


¢ Bits 51:MAXPHYADDR are reserved in every paging-structure entry. 

° The PS flag is reserved in a PML5E or a PML4E. 

* If 1-GByte pages are not supported, the PS flag is reserved in a PDPTE.! 

° Ifthe PS flag in a PDPTE is 1, bits 29:13 of the entry are reserved. 

° Ifthe PS flag in a PDE is 1, bits 20:13 of the entry are reserved. 

° If IA32_EFER.NXE = 0, the XD flag (bit 63) is reserved in every paging-structure entry. 


A reference using a linear address that is successfully translated to a physical address is performed only if allowed 
by the access rights of the translation; see Section 4.6. 


Figure 4-11 gives a summary of the formats of CR3 and the 4-level and 5-level paging-structure entries. For the 

paging structure entries, it identifies separately the format of entries that map pages, those that reference other 
paging structures, and those that do neither because they are “not present”; bit 0 (P) and bit 7 (PS) are highlighted 
because they determine how a paging-structure entry is used. 


Table 4-14. Format of a PML5 Entry (PML5€) that References a PML4 Table 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a PML4 table 

1 (R/W) Read/write; if 0, writes may not be allowed to the 256-TByte region controlled by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 256-TByte region controlled by this entry (see 
Section 4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the PML4 table referenced by this 
entry (see Section 4.9.2) 


1. See Section 4.1.4 for how to determine whether 1-GByte pages are supported. 
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Table 4-14. Format of a PML5 Entry (PML5€) that References a PML4 Table (Contd.) 


Bit Contents 

Position(s) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the PML4 table referenced by this 
entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether this entry has been used for linear-address translation (see Section 4.8) 

6 Ignored 

7 (PS) Reserved (must be 0) 

11:8 Ignored 

M-1:12 Physical address of 4-KByte aligned PML4 table referenced by this entry 

51:M Reserved (must be 0) 

62:52 Ignored 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 256-TByte region 
controlled by this entry; see Section 4.6); otherwise, reserved (must be 0) 


Table 4-15. Format of a PML4 Entry (PML4€) that References a Page-Directory-Pointer Table 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a page-directory-pointer table 

1 (R/W) Read/write; if 0, writes may not be allowed to the 512-GByte region controlled by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 512-GByte region controlled by this entry (see 
Section 4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page-directory-pointer table 


referenced by this entry (see Section 4.9.2) 


4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page-directory-pointer table 
referenced by this entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether this entry has been used for linear-address translation (see Section 4.8) 

6 Ignored 

7 (PS) Reserved (must be 0) 

11:8 Ignored 

M-1:12 Physical address of 4-KByte aligned page-directory-pointer table referenced by this entry 

51:M Reserved (must be 0) 

62:52 Ignored 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 512-GByte region 


controlled by this entry; see Section 4.6); otherwise, reserved (must be 0) 
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Table 4-16. Format of a Page-Directory-Pointer-Table Entry (PDPTE) that Maps a 1-GByte Page 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 1-GByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 1-GByte page referenced by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 1-GByte page referenced by this entry (see Section 
46) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 1-GByte page referenced by this 
entry (see Section 4.9.2) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 1-GByte page referenced by this 
entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether software has accessed the 1-GByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 1-GByte page referenced by this entry (see Section 4.8) 

7 (PS) Page size; must be 1 (otherwise, this entry references a page directory; see Table 4-17) 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 

11:9 Ignored 

12 (PAT) Pusey determines the memory type used to access the 1-GByte page referenced by this entry (see Section 
4.9.2) 

29:13 Reserved (must be 0) 

(M-1):30 Physical address of the 1-GByte page referenced by this entry 

51:M Reserved (must be 0) 

58:52 Ignored 

62:59 Protection key; if CR4.PKE = 1 or CR4.PKS = 1, this may control the page’s access rights (see Section 4.6.2); otherwise, 
it is not used to control access rights. 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 1-GByte page controlled by 
this entry; see Section 4.6); otherwise, reserved (must be 0) 

NOTES: 


1. The PAT is supported on all processors that support 4-level paging. 
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Table 4-17. Format of a Page-Directory-Pointer-Table Entry (PDPTE) that References a Page Directory 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a page directory 

1 (R/W) Read/write; if 0, writes may not be allowed to the 1-GByte region controlled by this entry (see Section 4.6) 

2 (U/S) 7 aad if O, user-mode accesses are not allowed to the 1-GByte region controlled by this entry (see Section 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page directory referenced by 
this entry (see Section 4.9.2) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page directory referenced by 
this entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether this entry has been used for linear-address translation (see Section 4.8) 

6 Ignored 

7 (PS) Page size; must be O (otherwise, this entry maps a 1-GByte page; see Table 4-16) 

11:8 Ignored 

(M-1):12 Physical address of 4-KByte aligned page directory referenced by this entry 

51:M Reserved (must be 0) 

62:52 Ignored 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 1-GByte region controlled 
by this entry; see Section 4.6); otherwise, reserved (must be 0) 

Table 4-18. Format of a Page-Directory Entry that Maps a 2-MByte Page 

Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 2-MByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 2-MByte page referenced by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 2-MByte page referenced by this entry (see Section 
4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 2-MByte page referenced by 
this entry (see Section 4.9.2) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 2-MByte page referenced by 
this entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether software has accessed the 2-MByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 2-MByte page referenced by this entry (see Section 4.8) 

7 (PS) Page size; must be 1 (otherwise, this entry references a page table; see Table 4-19) 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 
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Table 4-18. Format of a Page-Directory Entry that Maps a 2-MByte Page (Contd.) 
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Bit Contents 

Position(s) 

11:9 Ignored 

12 (PAT) Indirectly determines the memory type used to access the 2-MByte page referenced by this entry (see Section 
4.9.2) 

20:13 Reserved (must be 0) 

(M-1):21 Physical address of the 2-MByte page referenced by this entry 

51:M Reserved (must be 0) 

58:52 Ignored 

62:59 Protection key; if CR4.PKE = 1 or CR4.PKS = 1, this may control the page's access rights (see Section 4.6.2); 
otherwise, it is not used to control access rights. 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 2-MByte page controlled by 
this entry; see Section 4.6); otherwise, reserved (must be 0) 

Table 4-19. Format of a Page-Directory Entry that References a Page Table 

Bit Contents 

Position(s) 

0 (P) Present; must be 1 to reference a page table 

1 (R/W) Read/write; if 0, writes may not be allowed to the 2-MByte region controlled by this entry (see Section 4.6) 

2 (U/S) User/supervisor; if 0, user-mode accesses are not allowed to the 2-MByte region controlled by this entry (see Section 
4.6) 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the page table referenced by this 
entry (see Section 4.9.2) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the page table referenced by this 
entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether this entry has been used for linear-address translation (see Section 4.8) 

6 Ignored 

7 (PS) Page size; must be O (otherwise, this entry maps a 2-MByte page; see Table 4-18) 

11:8 Ignored 

(M-1):12 Physical address of 4-KByte aligned page table referenced by this entry 

51:M Reserved (must be 0) 

62:52 Ignored 

63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 2-MByte region controlled 
by this entry; see Section 4.6); otherwise, reserved (must be 0) 
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Table 4-20. Format of a Page-Table Entry that Maps a 4-KByte Page 


Bit Contents 

Position(s) 

0 (P) Present; must be 1 to map a 4-KByte page 

1 (R/W) Read/write; if 0, writes may not be allowed to the 4-KByte page referenced by this entry (see Section 4.6) 

2 (U/S) ae if O, user-mode accesses are not allowed to the 4-KByte page referenced by this entry (see Section 

3 (PWT) Page-level write-through; indirectly determines the memory type used to access the 4-KByte page referenced by 
this entry (see Section 4.9.2) 

4 (PCD) Page-level cache disable; indirectly determines the memory type used to access the 4-KByte page referenced by this 
entry (see Section 4.9.2) 

5 (A) Accessed; indicates whether software has accessed the 4-KByte page referenced by this entry (see Section 4.8) 

6 (D) Dirty; indicates whether software has written to the 4-KByte page referenced by this entry (see Section 4.8) 

7 (PAT) Indirectly determines the memory type used to access the 4-KByte page referenced by this entry (see Section 4.9.2) 

8 (G) Global; if CR4.PGE = 1, determines whether the translation is global (see Section 4.10); ignored otherwise 

11:9 Ignored 


(M-1):12 Physical address of the 4-KByte page referenced by this entry 


51:M Reserved (must be 0) 
58:52 Ignored 
62:59 Protection key; if CR4.PKE = 1 or CR4.PKS = 1, this may control the page's access rights (see Section 4.6.2); 


otherwise, it is not used to control access rights. 


63 (XD) If IA32_EFER.NXE = 1, execute-disable (if 1, instruction fetches are not allowed from the 4-KByte page controlled by 
this entry; see Section 4.6); otherwise, reserved (must be 0) 
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Figure 4-11. Formats of CR3 and Paging-Structure Entries with 4-Level Paging and 5-Level Paging 
NOTES: 
1. Mis an abbreviation for MAXPHYADDR. 
2. Reserved fields must be 0. 
3. If IA32_EFER.NXE = O and the P flag of a paging-structure entry is 1, the XD flag (bit 63) is reserved. 
4. The protection key is used only if software has enabled the appropriate feature; see Section 4.6.2. 


4-30 Vol. 3A 


PAGING 


4.6 ACCESS RIGHTS 


There is a translation for a linear address if the processes described in Section 4.3, Section 4.4.2, and Section 4.5 
(depending upon the paging mode) completes and produces a physical address. Whether an access is permitted by 
a translation is determined by the access rights specified by the paging-structure entries controlling the transla- 
tion;! paging-mode modifiers in CRO, CR4, and the IA32_EFER MSR; EFLAGS.AC; and the mode of the access. 


Section 4.6.1 describes how the processor determines the access rights for each linear address. Section 4.6.2 
provides additional information about how protection keys contribute to access-rights determination. (They do so 
only with 4-level paging and 5-level paging, and only if CR4.PKE = 1 or CR4.PKS = 1.) 


4.6.1 Determination of Access Rights 


Every access to a linear address is either a supervisor-mode access or a user-mode access. For all instruction 
fetches and most data accesses, this distinction is determined by the current privilege level (CPL): accesses made 
while CPL < 3 are supervisor-mode accesses, while accesses made while CPL = 3 are user-mode accesses. 


Some operations implicitly access system data structures with linear addresses; the resulting accesses to those 
data structures are supervisor-mode accesses regardless of CPL. Examples of such accesses include the following: 
accesses to the global descriptor table (GDT) or local descriptor table (LDT) to load a segment descriptor; accesses 
to the interrupt descriptor table (IDT) when delivering an interrupt or exception; and accesses to the task-state 
segment (TSS) as part of a task switch or change of CPL. All these accesses are called implicit supervisor-mode 
accesses regardless of CPL. Other accesses made while CPL < 3 are called explicit supervisor-mode accesses. 


Access rights are also controlled by the mode of a linear address as specified by the paging-structure entries 
controlling the translation of the linear address. If the U/S flag (bit 2) is 0 in at least one of the paging-structure 
entries, the address is a Ssupervisor-mode address. Otherwise, the address is a user-mode address. 


When the shadow -stack feature of control-flow enforcement technology (CET) is enabled, certain accesses to 
linear addresses are considered shadow-stack accesses (see Section 18.2, “Shadow Stacks”in the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 1). Like ordinary data accesses, each shadow-stack 
access is defined as being either a user access or a supervisor access. In general, a shadow-stack access is a user 
access if CPL = 3 and a supervisor access if CPL < 3. The WRUSS instruction is an exception; although it can be 
executed only if CPL = 0, the processor treats its shadow-stack accesses as user accesses. 


Shadow-stack accesses are allowed only to shadow-stack addresses. A linear address is a shadow-stack 
address if the following are true of the translation of the linear address: (1) the R/W flag (bit 1) is O and the dirty 
flag (bit 6) is 1 in the paging-structure entry that maps the page containing the linear address; and (2) the R/W 
flag is 1 in every other paging-structure entry controlling the translation of the linear address. 


The following items detail how paging determines access rights (only the items noted explicitly apply to shadow- 
stack accesses): 


¢ For supervisor-mode accesses: 


— Data may be read (implicitly or explicitly) from any supervisor-mode address with a protection key for 
which read access is permitted (see Section 4.6.2). 


— Data reads from user-mode pages. 
Access rights depend on the value of CR4.SMAP: 


¢ If CR4.SMAP = 0, data may be read from any user-mode address with a protection key for which read 
access is permitted (see Section 4.6.2). 


¢ If CR4.SMAP = 1, access rights depend on the value of EFLAGS.AC and whether the access is implicit or 
explicit: 


— If EFLAGS.AC = 1 and the access is explicit, data may be read from any user-mode address with a 
protection key for which read access is permitted (see Section 4.6.2). 


— If EFLAGS.AC = 0 or the access is implicit, data may not be read from any user-mode address. 


1. With PAE paging, the PDPTEs do not determine access rights. 
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— Data writes to supervisor-mode addresses. 
Access rights depend on the value of CRO.WP: 


If CRO.WP = 0, data may be written to any supervisor-mode address with a protection key for which 
write access is permitted (see Section 4.6.2). 


If CRO.WP = 1, data may be written to any supervisor-mode address with a translation for which the 
R/W flag (bit 1) is 1 in every paging-structure entry controlling the translation and with a protection key 
for which write access is permitted (see Section 4.6.2); data may not be written to any supervisor- 
mode address with a translation for which the R/W flag is 0 in any paging-structure entry controlling the 
translation. 


— Data writes to user-mode addresses. 
Access rights depend on the value of CRO.WP: 


If CRO.WP = 0, access rights depend on the value of CR4.SMAP: 


— If CR4.SMAP = 0, data may be written to any user-mode address with a protection key for which 
write access is permitted (see Section 4.6.2). 


— If CR4.SMAP = 1, access rights depend on the value of EFLAGS.AC and whether the access is 
implicit or explicit: 


e If EFLAGS.AC = 1 and the access is explicit, data may be written to any user-mode address 
with a protection key for which write access is permitted (see Section 4.6.2). 


e If EFLAGS.AC = 0 or the access is implicit, data may not be written to any user-mode address. 
If CRO.WP = 1, access rights depend on the value of CR4.SMAP: 


— If CR4.SMAP = 0, data may be written to any user-mode address with a translation for which the 
R/W flag is 1 in every paging-structure entry controlling the translation and with a protection key 
for which write access is permitted (see Section 4.6.2); data may not be written to any user-mode 
address with a translation for which the R/W flag is 0 in any paging-structure entry controlling the 
translation. 


— If CR4.SMAP = 1, access rights depend on the value of EFLAGS.AC and whether the access is 
implicit or explicit: 


e If EFLAGS.AC = 1 and the access is explicit, data may be written to any user-mode address 
with a translation for which the R/W flag is 1 in every paging-structure entry controlling the 
translation and with a protection key for which write access is permitted (see Section 4.6.2); 
data may not be written to any user-mode address with a translation for which the R/W flag is 
0 in any paging-structure entry controlling the translation. 


e If EFLAGS.AC = 0 or the access is implicit, data may not be written to any user-mode address. 


— Instruction fetches from supervisor-mode addresses. 


For 32-bit paging or if IA32_EFER.NXE = 0, instructions may be fetched from any supervisor-mode 
address. 


For other paging modes with IA32_EFER.NXE = 1, instructions may be fetched from any supervisor- 
mode address with a translation for which the XD flag (bit 63) is 0 in every paging-structure entry 
controlling the translation; instructions may not be fetched from any supervisor-mode address with a 
translation for which the XD flag is 1 in any paging-structure entry controlling the translation. 


— Instruction fetches from user-mode addresses. 
Access rights depend on the values of CR4.SMEP: 
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If CR4.SMEP = 0, access rights depend on the paging mode and the value of IA32_EFER.NXE: 


— For 32-bit paging or if IA32_EFER.NXE = 0, instructions may be fetched from any user-mode 
address. 


— For other paging modes with IA32_EFER.NXE = 1, instructions may be fetched from any user- 
mode address with a translation for which the XD flag is 0 in every paging-structure entry 
controlling the translation; instructions may not be fetched from any user-mode address with a 
translation for which the XD flag is 1 in any paging-structure entry controlling the translation. 
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¢ If CR4.SMEP = 1, instructions may not be fetched from any user-mode address. 


— Supervisor-mode shadow-stack accesses are allowed only to supervisor-mode shadow-stack addresses 
(see above). 


° For user-mode accesses: 


— Data reads. 
Access rights depend on the mode of the linear address: 


¢ Data may be read from any user-mode address with a protection key for which read access is permitted 
(see Section 4.6.2). 


¢ Data may not be read from any supervisor-mode address. 


— Data writes. 
Access rights depend on the mode of the linear address: 


* Data may be written to any user-mode address with a translation for which the R/W flag is 1 in every 
paging-structure entry controlling the translation and with a protection key for which write access is 
permitted (see Section 4.6.2). 


* Data may not be written to any supervisor-mode address. 


— Instruction fetches. 
Access rights depend on the mode of the linear address, the paging mode, and the value of 
IA32_EFER.NXE: 


¢ For 32-bit paging or if IA32_EFER.NXE = 0, instructions may be fetched from any user-mode address. 


¢ For other paging modes with IA32_EFER.NXE = 1, instructions may be fetched from any user-mode 
address with a translation for which the XD flag is 0 in every paging-structure entry controlling the 
translation. 


¢ Instructions may not be fetched from any supervisor-mode address. 


— User-mode shadow-stack accesses made outside enclave mode are allowed only to user-mode shadow- 
stack addresses (see above). User-mode shadow-stack accesses made in enclave mode are treated like 
ordinary data accesses (see above). 


A processor may cache information from the paging-structure entries in TLBs and paging-structure caches (see 
Section 4.10). These structures may include information about access rights. The processor may enforce access 
rights based on the TLBs and paging-structure caches instead of on the paging structures in memory. 


This fact implies that, if software modifies a paging-structure entry to change access rights, the processor might 
not use that change for a subsequent access to an affected linear address (see Section 4.10.4.3). See Section 
4.10.4.2 for how software can ensure that the processor uses the modified access rights. 


4.6.2 Protection Keys 


4-level paging and 5-level paging associate a 4-bit protection key with each linear address (the protection key 
located in bits 62:59 of the paging-structure entry that mapped the page containing the linear address; see Section 
4.5). Two protection key features control accesses to linear addresses based on their protection keys: 


° If CR4.PKE = 1, the PKRU register determines, for each protection key, whether user-mode addresses with that 
protection key may be read or written. 

° If CR4.PKS = 1, the IA32_PKRS MSR (MSR index 6E1H) determines, for each protection key, whether 
supervisor-mode addresses with that protection key may be read or written. 


32-bit paging and PAE paging do not associate linear addresses with protection keys. For the purposes of Section 
4.6.1, reads and writes are implicitly permitted for all protection keys with either of those paging modes. 


The PKRU register (protection-key rights for user pages) is a 32-bit register with the following format: for each / 
(0 <i < 15), PKRU[2/] is the access-disable bit for protection key i (AD/); PKRU[2/+1] is the write-disable bit 
for protection key / (WD/). The IA32_PKRS MSR has the same format (bits 63:32 of the MSR are reserved and must 
be zero). 
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Software can use the RDPKRU and WRPKRU instructions with ECX = 0 to read and write PKRU. In addition, the 
PKRU register is XSAVE-managed state and can thus be read and written by instructions in the XSAVE feature set. 
See Chapter 13, “Managing State Using the XSAVE Feature Set,” of Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1 for more information about the XSAVE feature set. 


Software can use the RDMSR and WRMSR instructions to read and write the IA32_PKRS MSR. Writes to the 
IA32_PKRS MSR using WRMSR are not serializing. The IA32_PKRS MSR is not XSAVE-managed. 


How a linear address’s protection key controls access to the address depends on the mode of the linear address: 


¢ A linear address’s protection key controls only data accesses to the address. It does not in any way affect 
instructions fetches from the address. 


¢ If CR4.PKE = 0, the protection key of a user-mode address does not control data accesses to the address (for 
the purposes of Section 4.6.1, reads and writes of user-mode addresses are implicitly permitted for all 
protection keys). 


If CR4.PKE = 1, use of the protection key / of a user-mode address depends on the value of the PKRU register: 
— If AD/ = 1, no data accesses are permitted. 
— If WD/i = 1, permission may be denied to certain data write accesses: 

e¢ User-mode write accesses are not permitted. 


¢ Supervisor-mode write accesses are not permitted if CRO.WP = 1. (If CRO.WP = 0, WDi does not affect 
supervisor-mode write accesses to user-mode addresses with protection key /.) 


° If CR4.PKS = 0, the protection key of a supervisor-mode address does not control data accesses to the address 
(for the purposes of Section 4.6.1, reads and writes of supervisor-mode addresses are implicitly permitted for 
all protection keys). 


If CR4.PKS = 1, use of the protection key i of a supervisor-mode address depends on the value of the 
IA32_PKRS MSR: 


— If AD/ = 1, no data accesses are permitted. 


— If WDi = 1, write accesses are not permitted if CRO.WP = 1. (If CRO.WP = 0, IA32_PKRS.WDi does not 
affect write accesses to supervisor-mode addresses with protection key /.) 


Protection keys apply to shadow-stack accesses just as they do to ordinary data accesses. 


4.7 PAGE-FAULT EXCEPTIONS 


Accesses using linear addresses may cause page-fault exceptions (#PF; exception 14). An access to a linear 
address may cause a page-fault exception for either of two reasons: (1) there is no translation for the linear 
address; or (2) there is a translation for the linear address, but its access rights do not permit the access. 


As noted in Section 4.3, Section 4.4.2, and Section 4.5, there is no translation for a linear address if the translation 
process for that address would use a paging-structure entry in which the P flag (bit 0) is O or one that sets a 
reserved bit. If there is a translation for a linear address, its access rights are determined as specified in Section 
4.6. 


When Intel® Software Guard Extensions (Intel® SGX) are enabled, the processor may deliver exception 14 for 
reasons unrelated to paging. See Section 33.3, “Access-control Requirements” and Section 33.20, “Enclave Page 
Cache Map (EPCM)” in Chapter 33, “Enclave Access Control and Data Structures.” Such an exception is called an 
SGX-induced page fault. The processor uses the error code to distinguish SGX-induced page faults from ordinary 
page faults. 
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Figure 4-12 illustrates the error code that the processor provides on delivery of a page-fault exception. The 
following items explain how the bits in the error code describe the nature of the page-fault exception: 
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P 0 The fault was caused by a non-present page. 
1 The fault was caused by a page-level protection violation. 


W/R 0 The access causing the fault was a read. 
1 The access causing the fault was a write. 


U/S 0 A supervisor-mode access caused the fault. 
A user-mode access caused the fault. 


= 


RSVD__O The fault was not caused by reserved bit violation. 
The fault was caused by a reserved bit set to 1 in some 
paging-structure entry. 


= 


/D 0 The fault was not caused by an instruction fetch. 
1 The fault was caused by an instruction fetch. 
PK 0 The fault was not caused by protection keys. 
1 There was a protection-key violation. 
ss The fault was not caused by a shadow-stack access. 


- Oo 


The fault was caused by a shadow-stack access. 


SGX 0 The fault is not related to SGX. 
1 The fault resulted from violation of SGX-specific access-control 
requirements. 


Figure 4-12. Page-Fault Error Code 


¢ =P flag (bit 0). 
This flag is O if there is no translation for the linear address because the P flag was 0 in one of the paging- 
structure entries used to translate that address. 

© ~W/R (bit 1). 
If the access causing the page-fault exception was a write, this flag is 1; otherwise, it is 0. This flag describes 
the access causing the page-fault exception, not the access rights specified by paging. 

° U/S (bit 2). 
If a user-mode access caused the page-fault exception, this flag is 1; it is 0 if a supervisor-mode access did so. 
This flag describes the access causing the page-fault exception, not the access rights specified by paging. User- 
mode and supervisor-mode accesses are defined in Section 4.6. 


¢ RSVD flag (bit 3). 
This flag is 1 if there is no translation for the linear address because a reserved bit was set in one of the paging- 
structure entries used to translate that address. (Because reserved bits are not checked in a paging-structure 
entry whose P flag is 0, bit 3 of the error code can be set only if bit 0 is also set.*) 


Bits reserved in the paging-structure entries are reserved for future functionality. Software developers should 
be aware that such bits may be used in the future and that a paging-structure entry that causes a page-fault 
exception on one processor might not do so in the future. 


1. Some past processors had errata for some page faults that occur when there is no translation for the linear address because the P 
flag was O in one of the paging-structure entries used to translate that address. Due to these errata, some such page faults pro- 
duced error codes that cleared bit O (P flag) and set bit 3 (RSVD flag). 
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¢ I/D flag (bit 4). 
This flag is 1 if (1) the access causing the page-fault exception was an instruction fetch; and (2) either 
(a) CR4.SMEP = 1; or (b) both (i) CR4.PAE = 1 (either PAE paging, 4-level paging, or 5-level paging is in use); 
and (ii) IA32_EFER.NXE = 1. Otherwise, the flag is 0. This flag describes the access causing the page-fault 
exception, not the access rights specified by paging. 


¢ PK flag (bit 5). 
This flag is 1 only for data accesses and only with 4-level paging and 5-level paging. In these cases, the setting 
depends on the mode of the address being accessed: 


— For accesses to supervisor-mode addresses, the flag is set if (1) CR4.PKS = 1; (2) the linear address has 
protection key /; and (3) the IA32_PKRS MSR (see Section 4.6.2) is such that either (a) AD/ = 1; or (b) the 
following all hold: (i) WD/ = 1; (ii) the access is a write access; and (iii) either CRO.WP = 1 or the access 
causing the page-fault exception was a user-mode access. (Note that this flag may be set on page faults 
due to user-mode accesses to supervisor-mode addresses.) 


— For accesses to user-mode addresses, the flag is set if (1) CR4.PKE = 1; (2) the linear address has 
protection key /; and (3) the PKRU register (see Section 4.6.2) is such that either (a) AD/ = 1; or (b) the 
following all hold: (i) WD/ = 1; (ii) the access is a write access; and (iii) either CRO.WP = 1 or the access 
causing the page-fault exception was a user-mode access. 


° SS (bit 1). 
If the access causing the page-fault exception was a shadow-stack access (including shadow-stack accesses in 
enclave mode), this flag is 1; otherwise, it is 0. This flag describes the access causing the page-fault exception, 
not the access rights specified by paging. 


° SGX flag (bit 15). 
This flag is 1 if the exception is unrelated to paging and resulted from violation of SGX-specific access-control 
requirements. Because such a violation can occur only if there is no ordinary page fault, this flag is set only if 
the P flag (bit 0) is 1 and the RSVD flag (bit 3) and the PK flag (bit 5) are both 0. 


Page-fault exceptions occur only due to an attempt to use a linear address. Failures to load the PDPTE registers 
with PAE paging (see Section 4.4.1) cause general-protection exceptions (#GP(0)) and not page-fault exceptions. 


4.8 ACCESSED AND DIRTY FLAGS 


For any paging-structure entry that is used during linear-address translation, bit 5 is the accessed flag.! For 
paging-structure entries that map a page (as opposed to referencing another paging structure), bit 6 is the dirty 
flag. These flags are provided for use by memory-management software to manage the transfer of pages and 
paging structures into and out of physical memory. 


Whenever the processor uses a paging-structure entry as part of linear-address translation, it sets the accessed 
flag in that entry (if it is not already set). 


Whenever there is a write to a linear address, the processor sets the dirty flag (if it is not already set) in the paging- 
structure entry that identifies the final physical address for the linear address (either a PTE or a paging-structure 
entry in which the PS flag is 1). 


NOTE 


If software on one logical processor writes to a page while software on another logical processor 
concurrently clears the R/W flag in the paging-structure entry that maps the page, execution on 
some processors may result in the entry’s dirty flag being set (due to the write on the first logical 
processor) and the entry’s R/W flag being clear (due to the update to the entry on the second 
logical processor). This will never occur on a processor that supports control-flow enforcement 
technology (CET). Specifically, a processor that supports CET will never set the dirty flag ina 
paging-structure entry in which the R/W flag is clear. 


1. With PAE paging, the PDPTEs are not used during linear-address translation but only to load the PDPTE registers for some execu- 
tions of the MOV CR instruction (see Section 4.4.1). For this reason, the PDPTEs do not contain accessed flags with PAE paging. 
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Memory-management software may clear these flags when a page or a paging structure is initially loaded into 
physical memory. These flags are “sticky,” meaning that, once set, the processor does not clear them; only soft- 
ware can clear them. 


A processor may cache information from the paging-structure entries in TLBs and paging-structure caches (see 
Section 4.10). This fact implies that, if software changes an accessed flag or a dirty flag from 1 to 0, the processor 
might not set the corresponding bit in memory on a subsequent access using an affected linear address (see 
Section 4.10.4.3). See Section 4.10.4.2 for how software can ensure that these bits are updated as desired. 


NOTE 


The accesses used by the processor to set these flags may or may not be exposed to the 
processor’s self-modifying code detection logic. If the processor is executing code from the same 
memory area that is being used for the paging structures, the setting of these flags may or may not 
result in an immediate change to the executing code stream. 


4.9 PAGING AND MEMORY TYPING 


The memory type of a memory access refers to the type of caching used for that access. Chapter 11, “Memory 
Cache Control” provides many details regarding memory typing in the Intel-64 and IA-32 architectures. This 
section describes how paging contributes to the determination of memory typing. 


The way in which paging contributes to memory typing depends on whether the processor supports the Page 
Attribute Table (PAT; see Section 11.12).! Section 4.9.1 and Section 4.9.2 explain how paging contributes to 
memory typing depending on whether the PAT is supported. 


4.9.1 Paging and Memory Typing When the PAT is Not Supported (Pentium Pro and 
Pentium II Processors) 


NOTE 


The PAT is supported on all processors that support 4-level paging or 5-level paging. Thus, this 
section applies only to 32-bit paging and PAE paging. 


If the PAT is not supported, paging contributes to memory typing in conjunction with the memory-type range regis- 
ters (MTRRs) as specified in Table 11-6 in Section 11.5.2.1. 


For any access to a physical address, the table combines the memory type specified for that physical address by 
the MTRRs with a PCD value and a PWT value. The latter two values are determined as follows: 


¢ For an access to a PDE with 32-bit paging, the PCD and PWT values come from CR3. 
¢ For an access to a PDE with PAE paging, the PCD and PWT values come from the relevant PDPTE register. 
° For an access to a PTE, the PCD and PWT values come from the relevant PDE. 


¢ For an access to the physical address that is the translation of a linear address, the PCD and PWT values come 
from the relevant PTE (if the translation uses a 4-KByte page) or the relevant PDE (otherwise). 


¢ With PAE paging, the UC memory type is used when loading the PDPTEs (see Section 4.4.1). 


4.9.2 Paging and Memory Typing When the PAT is Supported (Pentium III and More Recent 
Processor Families) 


If the PAT is supported, paging contributes to memory typing in conjunction with the PAT and the memory-type 
range registers (MTRRs) as specified in Table 11-7 in Section 11.5.2.2. 


1. The PAT is supported on Pentium Ill and more recent processor families. See Section 4.1.4 for how to determine whether the PAT is 
supported. 
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The PAT is a 64-bit MSR (IA32_PAT; MSR index 277H) comprising eight (8) 8-bit entries (entry / comprises 
bits 8/+7:8/ of the MSR). 


For any access to a physical address, the table combines the memory type specified for that physical address by the 
MTRRs with a memory type selected from the PAT. Table 11-11 in Section 11.12.3 specifies how a memory type is 
selected from the PAT. Specifically, it comes from entry / of the PAT, where / is defined as follows: 


¢ For an access to an entry in a paging structure whose address is in CR3 (e.g., the PML4 table with 4-level 
paging): 
— For 4-level paging or 5-level paging with CR4.PCIDE = 1, / = 0. 
— Otherwise, i = 2*PCD+PWT, where the PCD and PWT values come from CR3. 


¢ For an access to a PDE with PAE paging, i = 2*PCD+PWT, where the PCD and PWT values come from the 
relevant PDPTE register. 


° For an access to a paging-structure entry X whose address is in another paging-structure entry Y, / = 
2*PCD+PWT, where the PCD and PWT values come from Y. 


° For an access to the physical address that is the translation of a linear address, / = 4*PAT+2*PCD+PWT, where 
the PAT, PCD, and PWT values come from the relevant PTE (if the translation uses a 4-KByte page), the relevant 
PDE (if the translation uses a 2-MByte page or a 4-MByte page), or the relevant PDPTE (if the translation uses 
a 1-GByte page). 


* With PAE paging, the WB memory type is used when loading the PDPTEs (see Section 4.4.1).! 


4.9.3 Caching Paging-Related Information about Memory Typing 


A processor may cache information from the paging-structure entries in TLBs and paging-structure caches (see 
Section 4.10). These structures may include information about memory typing. The processor may use memory- 
typing information from the TLBs and paging-structure caches instead of from the paging structures in memory. 


This fact implies that, if software modifies a paging-structure entry to change the memory-typing bits, the 
processor might not use that change for a subsequent translation using that entry or for access to an affected linear 
address. See Section 4.10.4.2 for how software can ensure that the processor uses the modified memory typing. 


4.10 CACHING TRANSLATION INFORMATION 


The Intel-64 and IA-32 architectures may accelerate the address-translation process by caching data from the 
paging structures on the processor. Because the processor does not ensure that the data that it caches are always 
consistent with the structures in memory, it is important for software developers to understand how and when the 
processor may cache such data. They should also understand what actions software can take to remove cached 
data that may be inconsistent and when it should do so. This section provides software developers information 
about the relevant processor operation. 


Section 4.10.1 introduces process-context identifiers (PCIDs), which a logical processor may use to distinguish 
information cached for different linear-address spaces. Section 4.10.2 and Section 4.10.3 describe how the 
processor may cache information in translation lookaside buffers (TLBs) and paging-structure caches, respectively. 
Section 4.10.4 explains how software can remove inconsistent cached information by invalidating portions of the 
TLBs and paging-structure caches. Section 4.10.5 describes special considerations for multiprocessor systems. 


4.10.1 Process-Context Identifiers (PCIDs) 


Process-context identifiers (PCIDs) are a facility by which a logical processor may cache information for multiple 
linear-address spaces. The processor may retain cached information when software switches to a different linear- 
address space with a different PCID (e.g., by loading CR3; see Section 4.10.4.1 for details). 


1. Some older IA-32 processors used the UC memory type when loading the PDPTEs. Some processors may use the UC memory type if 
CRO.CD = 1 or if the MTRRs are disabled. These behaviors are model-specific and not architectural. 
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A PCID is a 12-bit identifier. Non-zero PCIDs are enabled by setting the PCIDE flag (bit 17) of CR4. If CR4.PCIDE = 
0, the current PCID is always 000H; otherwise, the current PCID is the value of bits 11:0 of CR3. Not all processors 
allow CR4.PCIDE to be set to 1; see Section 4.1.4 for how to determine whether this is allowed. 


The processor ensures that CR4.PCIDE can be 1 only in IA-32e mode (thus, 32-bit paging and PAE paging use only 
PCID O0OH). In addition, software can change CR4.PCIDE from 0 to 1 only if CR3[11:0] = OOOH. These require- 
ments are enforced by the following limitations on the MOV CR instruction: 


¢ MOV to CR4 causes a general-protection exception (#GP) if it would change CR4.PCIDE from 0 to 1 and either 
IA32_EFER.LMA = 0 or CR3[11:0] # OOOH. 


¢ MOV to CRO causes a general-protection exception if it would clear CRO.PG to 0 while CR4.PCIDE = 1. 


When a logical processor creates entries in the TLBs (Section 4.10.2) and paging-structure caches (Section 
4.10.3), it associates those entries with the current PCID. When using entries in the TLBs and paging-structure 
caches to translate a linear address, a logical processor uses only those entries associated with the current PCID 
(see Section 4.10.2.4 for an exception). 


If CR4.PCIDE = 0, a logical processor does not cache information for any PCID other than OOOH. This is because 
(1) if CR4.PCIDE = 0, the logical processor will associate any newly cached information with the current PCID, 
OOOH; and (2) if MOV to CR4 clears CR4.PCIDE, all cached information is invalidated (see Section 4.10.4.1). 


NOTE 


In revisions of this manual that were produced when no processors allowed CR4.PCIDE to be set to 
1, Section 4.10 discussed the caching of translation information without any reference to PCIDs. 
While the section now refers to PCIDs in its specification of this caching, this documentation change 
is not intended to imply any change to the behavior of processors that do not allow CR4.PCIDE to 
be set to 1. 


4.10.2 | Translation Lookaside Buffers (TLBs) 


A processor may cache information about the translation of linear addresses in translation lookaside buffers 
(TLBs). In general, TLBs contain entries that map page numbers to page frames; these terms are defined in 
Section 4.10.2.1. Section 4.10.2.2 describes how information may be cached in TLBs, and Section 4.10.2.3 gives 
details of TLB usage. Section 4.10.2.4 explains the global-page feature, which allows software to indicate that 
certain translations should receive special treatment when cached in the TLBs. 


4.10.2.1 Page Numbers, Page Frames, and Page Offsets 


Section 4.3, Section 4.4.2, and Section 4.5 give details of how the different paging modes translate linear 
addresses to physical addresses. Specifically, the upper bits of a linear address (called the page number) deter- 
mine the upper bits of the physical address (called the page frame); the lower bits of the linear address (called 
the page offset) determine the lower bits of the physical address. The boundary between the page number and 
the page offset is determined by the page size. Specifically: 


¢ 32-bit paging: 


— Ifthe translation does not use a PTE (because CR4.PSE = 1 and the PS flag is 1 in the PDE used), the page 
size is 4 MBytes and the page number comprises bits 31:22 of the linear address. 


— Ifthe translation does use a PTE, the page size is 4 KBytes and the page number comprises bits 31:12 of 
the linear address. 


° PAE paging: 


— If the translation does not use a PTE (because the PS flag is 1 in the PDE used), the page size is 2 MBytes 
and the page number comprises bits 31:21 of the linear address. 


— Ifthe translation does use a PTE, the page size is 4 KBytes and the page number comprises bits 31:12 of 
the linear address. 


° = 4-level paging and 5-level paging: 
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— Ifthe translation does not use a PDE (because the PS flag is 1 in the PDPTE used), the page size is 1 GByte 
and the page number comprises bits 47:30 of the linear address. 


— Ifthe translation does use a PDE but does not uses a PTE (because the PS flag is 1 in the PDE used), the 
page size is 2 MBytes and the page number comprises bits 47:21 of the linear address. 


— Ifthe translation does use a PTE, the page size is 4 KBytes and the page number comprises bits 47:12 of 
the linear address. 


4.10.2.2 Caching Translations in TLBs 


The processor may accelerate the paging process by caching individual translations in translation lookaside 
buffers (TLBs). Each entry in a TLB is an individual translation. Each translation is referenced by a page number. 
It contains the following information from the paging-structure entries used to translate linear addresses with the 
page number: 


¢ The physical address corresponding to the page number (the page frame). 


° The access rights from the paging-structure entries used to translate linear addresses with the page number 
(see Section 4.6): 


— The logical-AND of the R/W flags. 

— The logical-AND of the U/S flags. 

— The logical-OR of the XD flags (necessary only if IA32_EFER.NXE = 1). 
— The protection key (only with 4-level paging and 5-level paging). 


e Attributes from a paging-structure entry that identifies the final page frame for the page number (either a PTE 
or a paging-structure entry in which the PS flag is 1): 


— The dirty flag (see Section 4.8). 
— The memory type (see Section 4.9). 


(TLB entries may contain other information as well. A processor may implement multiple TLBs, and some of these 
may be for special purposes, e.g., only for instruction fetches. Such special-purpose TLBs may not contain some of 
this information if it is not necessary. For example, a TLB used only for instruction fetches need not contain infor- 
mation about the R/W and dirty flags.) 


As noted in Section 4.10.1, any TLB entries created by a logical processor are associated with the current PCID. 


Processors need not implement any TLBs. Processors that do implement TLBs may invalidate any TLB entry at any 
time. Software should not rely on the existence of TLBs or on the retention of TLB entries. 


4.10.2.3 Details of TLB Use 


Because the TLBs cache entries only for linear addresses with translations, there can be a TLB entry for a page 
number only if the P flag is 1 and the reserved bits are 0 in each of the paging-structure entries used to translate 
that page number. In addition, the processor does not cache a translation for a page number unless the accessed 
flag is 1 in each of the paging-structure entries used during translation; before caching a translation, the processor 
sets any of these accessed flags that is not already 1. 


Subject to the limitations given in the previous paragraph, the processor may cache a translation for any linear 
address, even if that address is not used to access memory. For example, the processor may cache translations 
required for prefetches and for accesses that result from speculative execution that would never actually occur in 
the executed code path. 


If the page number of a linear address corresponds to a TLB entry associated with the current PCID, the processor 
may use that TLB entry to determine the page frame, access rights, and other attributes for accesses to that linear 
address. In this case, the processor may not actually consult the paging structures in memory. The processor may 
retain a TLB entry unmodified even if software subsequently modifies the relevant paging-structure entries in 
memory. See Section 4.10.4.2 for how software can ensure that the processor uses the modified paging-structure 
entries. 


If the paging structures specify a translation using a page larger than 4 KBytes, some processors may cache 
multiple smaller-page TLB entries for that translation. Each such TLB entry would be associated with a page 
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number corresponding to the smaller page size (e.g., bits 47:12 of a linear address with 4-level paging), even 
though part of that page number (e.g., bits 20:12) is part of the offset with respect to the page specified by the 
paging structures. The upper bits of the physical address in such a TLB entry are derived from the physical address 
in the PDE used to create the translation, while the lower bits come from the linear address of the access for which 
the translation is created. There is no way for software to be aware that multiple translations for smaller pages 
have been used for a large page. For example, an execution of INVLPG for a linear address on such a page invali- 
dates any and all smaller-page TLB entries for the translation of any linear address on that page. 


If software modifies the paging structures so that the page size used for a 4-KByte range of linear addresses 
changes, the TLBs may subsequently contain multiple translations for the address range (one for each page size). 
A reference to a linear address in the address range may use any of these translations. Which translation is used 
may vary from one execution to another, and the choice may be implementation-specific. 


4.10.2.4 Global Pages 


The Intel-64 and IA-32 architectures also allow for global pages when the PGE flag (bit 7) is 1 in CR4. If the G flag 
(bit 8) is 1 in a paging-structure entry that maps a page (either a PTE or a paging-structure entry in which the PS 
flag is 1), any TLB entry cached for a linear address using that paging-structure entry is considered to be global. 
Because the G flag is used only in paging-structure entries that map a page, and because information from such 
entries is not cached in the paging-structure caches, the global-page feature does not affect the behavior of the 
paging-structure caches. 


A logical processor may use a global TLB entry to translate a linear address, even if the TLB entry is associated with 
a PCID different from the current PCID. 


4.10.3. Paging-Structure Caches 


In addition to the TLBs, a processor may cache other information about the paging structures in memory. 


4.10.3.1 Caches for Paging Structures 
A processor may support any or all of the following paging-structure caches: 


¢ PMLSE cache (5-level paging only). Each PML5E-cache entry is referenced by a 9-bit value and is used for 
linear addresses for which bits 56:40 have that value. The entry contains information from the PML5E used to 
translate such linear addresses: 


— The physical address from the PML5E (the address of the PML4 table). 
— The value of the R/W flag of the PML5E. 

— The value of the U/S flag of the PML5E. 

— The value of the XD flag of the PML5E. 

— The values of the PCD and PWT flags of the PML5E. 

The following items detail how a processor may use the PMLSE cache: 


— If the processor has a PML5E-cache entry for a linear address, it may use that entry when translating the 
linear address (instead of the PMLSE in memory). 


— The processor does not create a PMLSE-cache entry unless the P flag is 1 and all reserved bits are 0 in the 
PML5E in memory. 


— The processor does not create a PMLSE-cache entry unless the accessed flag is 1 in the PMLS5E in memory; 
before caching a translation, the processor sets the accessed flag if it is not already 1. 


— The processor may create a PML5E-cache entry even if there are no translations for any linear address that 
might use that entry (e.g., because the P flags are 0 in all entries in the referenced PML4 table). 


— Ifthe processor creates a PML5E-cache entry, the processor may retain it unmodified even if software 
subsequently modifies the corresponding PML5E in memory. 


° PML4E cache (4-level paging and 5-level paging only). The use of the PML4E cache depends on the paging 
mode: 
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For 4-level paging, each PML4E-cache entry is referenced by a 9-bit value and is used for linear addresses 
for which bits 47:39 have that value. 


For 5-level paging, each PML4E-cache entry is referenced by an 18-bit value and is used for linear 
addresses for which bits 56:39 have that value. 


A PML4E-cache entry contains information from the PML5E and PML4E used to translate the relevant linear 
addresses (for 4-level paging, the PML5E does not apply): 


The physical address from the PML4E (the address of the page-directory-pointer table). 
The logical-AND of the R/W flags in the PML5E and the PML4E. 

The logical-AND of the U/S flags in the PML5E and the PML4E. 

The logical-OR of the XD flags in the PML5E and the PML4E. 

The values of the PCD and PWT flags of the PML4E. 


The following items detail how a processor may use the PML4E cache: 


If the processor has a PML4E-cache entry for a linear address, it may use that entry when translating the 
linear address (instead of the PML5E and PML4E in memory). 


The processor does not create a PML4E-cache entry unless the P flags are 1 and all reserved bits are 0 in 
the PML5E and the PML4E in memory. 


The processor does not create a PML4E-cache entry unless the accessed flags are 1 in the PML5E and the 
PML4E in memory; before caching a translation, the processor sets any accessed flags that are not already 
1. 


The processor may create a PML4E-cache entry even if there are no translations for any linear address that 
might use that entry (e.g., because the P flags are 0 in all entries in the referenced page-directory-pointer 
table). 


If the processor creates a PML4E-cache entry, the processor may retain it unmodified even if software 
subsequently modifies the corresponding PML4E in memory. 


* PDPTE cache (4-level paging and 5-level paging only).! The use of the PML4E cache depends on the paging 
mode: 


For 4-level paging, each PDPTE-cache entry is referenced by an 18-bit value and is used for linear 
addresses for which bits 47:30 have that value. 


For 5-level paging, each PDPTE-cache entry is referenced by a 27-bit value and is used for linear addresses 
for which bits 56:30 have that value. 


A PDPTE-cache entry contains information from the PML5E, PML4E, PDPTE used to translate the relevant linear 
addresses (for 4-level paging, the PML5E does not apply): 


The physical address from the PDPTE (the address of the page directory). (No PDPTE-cache entry is created 
for a PDPTE that maps a 1-GByte page.) 


The logical-AND of the R/W flags in the PML5E, PML4E, and PDPTE. 
The logical-AND of the U/S flags in the PML5E, PML4E, and PDPTE. 
The logical-OR of the XD flags in the PML5E, PML4E, and PDPTE. 
The values of the PCD and PWT flags of the PDPTE. 


The following items detail how a processor may use the PDPTE cache: 


If the processor has a PDPTE-cache entry for a linear address, it may use that entry when translating the 
linear address (instead of the PML5E, PML4E, and PDPTE in memory). 


The processor does not create a PDPTE-cache entry unless the P flags are 1, the PS flags are 0, and the 
reserved bits are 0 in the PMLSE, PML4E, and PDPTE in memory. 


1. With PAE paging, the PDPTEs are stored in internal, non-architectural registers. The operation of these registers is described in Sec- 
tion 4.4.1 and differs from that described here. 
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The processor does not create a PDPTE-cache entry unless the accessed flags are 1 in the PMLSE, PML4E 
and PDPTE in memory; before caching a translation, the processor sets any accessed flags that are not 
already 1. 


The processor may create a PDPTE-cache entry even if there are no translations for any linear address that 
might use that entry. 


If the processor creates a PDPTE-cache entry, the processor may retain it unmodified even if software 
subsequently modifies the corresponding PMLS5E, PML4E, or PDPTE in memory. 


© PDE cache. The use of the PDE cache depends on the paging mode: 


For 32-bit paging, each PDE-cache entry is referenced by a 10-bit value and is used for linear addresses for 
which bits 31:22 have that value. 


For PAE paging, each PDE-cache entry is referenced by an 11-bit value and is used for linear addresses for 
which bits 31:21 have that value. 


For 4-level paging, each PDE-cache entry is referenced by a 27-bit value and is used for linear addresses 
for which bits 47:21 have that value. 


For 5-level paging, each PDE-cache entry is referenced by a 36-bit value and is used for linear addresses 
for which bits 56:21 have that value. 


A PDE-cache entry contains information from the PML5E, PML4E, PDPTE, and PDE used to translate the relevant 
linear addresses (for 32-bit paging and PAE paging, only the PDE applies; for 4-level paging, the PML5E does 
not apply): 


The physical address from the PDE (the address of the page table). (No PDE-cache entry is created for a 
PDE that maps a page.) 


The logical-AND of the R/W flags in the PML5E, PML4E, PDPTE, and PDE. 
The logical-AND of the U/S flags in the PML5E, PML4E, PDPTE, and PDE. 
The logical-OR of the XD flags in the PML5SE, PML4E, PDPTE, and PDE. 
The values of the PCD and PWT flags of the PDE. 


The following items detail how a processor may use the PDE cache (references below to PML5Es, PML4Es, and 
PDPTEs apply only to 4-level paging and to 5-level paging, as appropriate): 


If the processor has a PDE-cache entry for a linear address, it may use that entry when translating the 
linear address (instead of the PMLSE, PML4E, PDPTE, and PDE in memory). 


The processor does not create a PDE-cache entry unless the P flags are 1, the PS flags are 0, and the 
reserved bits are 0 in the PMLSE, PML4E, PDPTE, and PDE in memory. 


The processor does not create a PDE-cache entry unless the accessed flag is 1 in the PML5E, PML4E, PDPTE, 
and PDE in memory; before caching a translation, the processor sets any accessed flags that are not 
already 1. 


The processor may create a PDE-cache entry even if there are no translations for any linear address that 
might use that entry. 


If the processor creates a PDE-cache entry, the processor may retain it unmodified even if software subse- 
quently modifies the corresponding PML5E, PML4E, PDPTE, or PDE in memory. 


Information from a paging-structure entry can be included in entries in the paging-structure caches for other 
paging-structure entries referenced by the original entry. For example, if the R/W flag is 0 ina PML4E, then the R/W 
flag will be 0 in any PDPTE-cache entry for a PDPTE from the page-directory-pointer table referenced by that 
PML4E. This is because the R/W flag of each such PDPTE-cache entry is the logical-AND of the R/W flags in the 
appropriate PML4E and PDPTE. 


The paging-structure caches contain information only from paging-structure entries that reference other paging 
structures (and not those that map pages). Because the G flag is not used in such paging-structure entries, the 
global-page feature does not affect the behavior of the paging-structure caches. 


The processor may create entries in paging-structure caches for translations required for prefetches and for 
accesses that are a result of speculative execution that would never actually occur in the executed code path. 
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As noted in Section 4.10.1, any entries created in paging-structure caches by a logical processor are associated 
with the current PCID. 


A processor may or may not implement any of the paging-structure caches. Software should rely on neither their 
presence nor their absence. The processor may invalidate entries in these caches at any time. Because the 
processor may create the cache entries at the time of translation and not update them following subsequent modi- 
fications to the paging structures in memory, software should take care to invalidate the cache entries appropri- 
ately when causing such modifications. The invalidation of TLBs and the paging-structure caches is described in 
Section 4.10.4. 


4.10.3.2 Using the Paging-Structure Caches to Translate Linear Addresses 


When a linear address is accessed, the processor uses a procedure such as the following to determine the physical 
address to which it translates and whether the access should be allowed: 


e If the processor finds a TLB entry that is for the page number of the linear address and that is associated with 
the current PCID (or which is global), it may use the physical address, access rights, and other attributes from 
that entry. 


e If the processor does not find a relevant TLB entry, it may use the upper bits of the linear address to select an 
entry from the PDE cache that is associated with the current PCID (Section 4.10.3.1 indicates which bits are 
used in each paging mode). It can then use that entry to complete the translation process (locating a PTE, etc.) 
as if it had traversed the PDE (and, for 4-level paging and 5-level paging, the PDPTE, PML4E, and PML5E, as 
appropriate) corresponding to the PDE-cache entry. 


¢ The following items apply when 4-level paging or 5-level paging is used: 


— Ifthe processor does not find a relevant TLB entry or PDE-cache entry, it may use the upper bits of the 
linear address (for 4-level paging, bits 47:30; for 5-level paging, bits 56:30) to select an entry from the 
PDPTE cache that is associated with the current PCID. It can then use that entry to complete the translation 
process (locating a PDE, etc.) as if it had traversed the PDPTE, the PML4E, and (for 5-level paging) the 
PMLS5E corresponding to the PDPTE-cache entry. 


— If the processor does not find a relevant TLB entry, PDE-cache entry, or PDPTE-cache entry, it may use the 
upper bits of the linear address (for 4-level paging, bits 47:39; for 5-level paging, bits 56:39) to select an 
entry from the PML4E cache that is associated with the current PCID. It can then use that entry to complete 
the translation process (locating a PDPTE, etc.) as if it had traversed the corresponding PML4E. 


— With 5-level paging, if the processor does not find a relevant TLB entry, PDE-cache entry, PDPTE-cache 
entry, or PML4E-cache entry, it may use bits 56:48 of the linear address to select an entry from the PMLS5SE 
cache that is associated with the current PCID. It can then use that entry to complete the translation 
process (locating a PML4E, etc.) as if it had traversed the corresponding PML5E. 


(Any of the above steps would be skipped if the processor does not support the cache in question.) 


If the processor does not find a TLB or paging-structure-cache entry for the linear address, it uses the linear 
address to traverse the entire paging-structure hierarchy, as described in Section 4.3, Section 4.4.2, and Section 
4.5. 


4.10.3.3. Multiple Cached Entries for a Single Paging-Structure Entry 


The paging-structure caches and TLBs may contain multiple entries associated with a single PCID and with infor- 
mation derived from a single paging-structure entry. The following items give some examples for 4-level paging: 


° Suppose that two PML4Es contain the same physical address and thus reference the same page-directory- 
pointer table. Any PDPTE in that table may result in two PDPTE-cache entries, each associated with a different 
set of linear addresses. Specifically, suppose that the nt and nth entries in the PML4 table contain the same 
physical address. This implies that the physical address in the m™ PDPTE in the page-directory-pointer table 
would appear in the PDPTE-cache entries associated with both p,; and p>, where (p; » 9) = ny, (p2 » 9) = No, 
and (p; & 1FFH) = (p> & 1FFH) = m. This is because both PDPTE-cache entries use the same PDPTE, one 
resulting from a reference from the n,"" PML4E and one from the n3"" PML4E. 


¢ Suppose that the first PML4E (i.e., the one in position 0) contains the physical address X in CR3 (the physical 
address of the PML4 table). This implies the following: 
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— Any PML4-cache entry associated with linear addresses with 0 in bits 47:39 contains address X. 


— Any PDPTE-cache entry associated with linear addresses with 0 in bits 47:30 contains address X. This is 
because the translation for a linear address for which the value of bits 47:30 is 0 uses the value of 
bits 47:39 (0) to locate a page-directory-pointer table at address X (the address of the PML4 table). It then 
uses the value of bits 38:30 (also 0) to find address X again and to store that address in the PDPTE-cache 
entry. 


— Any PDE-cache entry associated with linear addresses with 0 in bits 47:21 contains address X for similar 
reasons. 


— Any TLB entry for page number 0 (associated with linear addresses with 0 in bits 47:12) translates to page 
frame X » 12 for similar reasons. 


The same PML4E contributes its address X to all these cache entries because the self-referencing nature of the 
entry causes it to be used as a PML4E, a PDPTE, a PDE, and a PTE. 


4.10.4 _ Invalidation of TLBs and Paging-Structure Caches 


As noted in Section 4.10.2 and Section 4.10.3, the processor may create entries in the TLBs and the paging-struc- 
ture caches when linear addresses are translated, and it may retain these entries even after the paging structures 
used to create them have been modified. To ensure that linear-address translation uses the modified paging struc- 
tures, software should take action to invalidate any cached entries that may contain information that has since 
been modified. 


4.10.4.1 Operations that Invalidate TLBs and Paging-Structure Caches 


The following instructions invalidate entries in the TLBs and the paging-structure caches: 


INVLPG. This instruction takes a single operand, which is a linear address. The instruction invalidates any TLB 
entries that are for a page number corresponding to the linear address and that are associated with the current 
PCID. It also invalidates any global TLB entries with that page number, regardless of PCID (see Section 
4.10.2.4).! INVLPG also invalidates all entries in all paging-structure caches associated with the current PCID, 
regardless of the linear addresses to which they correspond. 


INVPCID. The operation of this instruction is based on instruction operands, called the INVPCID type and the 
INVPCID descriptor. Four INVPCID types are currently defined: 


— Individual-address. If the INVPCID type is 0, the logical processor invalidates mappings—except global 
translations—associated with the PCID specified in the INVPCID descriptor and that would be used to 
translate the linear address specified in the INVPCID descriptor.? (The instruction may also invalidate global 
translations, as well as mappings associated with other PCIDs and for other linear addresses. ) 


— Single-context. If the INVPCID type is 1, the logical processor invalidates all mappings—except global 
translations—associated with the PCID specified in the INVPCID descriptor. (The instruction may also 
invalidate global translations, as well as mappings associated with other PCIDs.) 


— All-context, including globals. If the INVPCID type is 2, the logical processor invalidates 
mappings—including global translations—associated with all PCIDs. 


— All-context. If the INVPCID type is 3, the logical processor invalidates mappings—except global transla- 
tions—associated with all PCIDs. (The instruction may also invalidate global translations. ) 


See Chapter 3 of the Intel 64 and IA-32 Architecture Software Developer’s Manual, Volume 2A for details of the 
INVPCID instruction. 


MOV to CRO. The instruction invalidates all TLB entries (including global entries) and all entries in all paging- 
structure caches (for all PCIDs) if it changes the value of CRO.PG from 1 to 0. 


MOV to CR3. The behavior of the instruction depends on the value of CR4.PCIDE: 


If the paging structures map the linear address using a page larger than 4 KBytes and there are multiple TLB entries for that page 
(see Section 4.10.2.3), the instruction invalidates all of them. 


If the paging structures map the linear address using a page larger than 4 KBytes and there are multiple TLB entries for that page 
(see Section 4.10.2.3), the instruction invalidates all of them. 
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— If CR4.PCIDE = 0, the instruction invalidates all TLB entries associated with PCID O00H except those for 
global pages. It also invalidates all entries in all paging-structure caches associated with PCID OOOH. 


— If CR4.PCIDE = 1 and bit 63 of the instruction’s source operand is 0, the instruction invalidates all TLB 
entries associated with the PCID specified in bits 11:0 of the instruction’s source operand except those for 
global pages. It also invalidates all entries in all paging-structure caches associated with that PCID. It is not 
required to invalidate entries in the TLBs and paging-structure caches that are associated with other PCIDs. 


— If CR4.PCIDE = 1 and bit 63 of the instruction’s source operand is 1, the instruction is not required to 
invalidate any TLB entries or entries in paging-structure caches. 

MOV to CR4. The behavior of the instruction depends on the bits being modified: 

— The instruction invalidates all TLB entries (including global entries) and all entries in all paging-structure 


caches (for all PCIDs) if (1) it changes the value of CR4.PGE;! or (2) it changes the value of the CR4.PCIDE 
from 1 to 0. 


— The instruction invalidates all TLB entries and all entries in all paging-structure caches for the current PCID 
if (1) it changes the value of CR4.PAE; or (2) it changes the value of CR4.SMEP from 0 to 1. 


Task switch. If a task switch changes the value of CR3, it invalidates all TLB entries associated with PCID O000H 
except those for global pages. It also invalidates all entries in all paging-structure caches associated with PCID 
OOOH. 


VMX transitions. See Section 4.11.1. 


The processor is always free to invalidate additional entries in the TLBs and paging-structure caches. The following 
are some examples: 


INVLPG may invalidate TLB entries for pages other than the one corresponding to its linear-address operand. It 
may invalidate TLB entries and paging-structure-cache entries associated with PCIDs other than the current 
PCID. 


INVPCID may invalidate TLB entries for pages other than the one corresponding to the specified linear address. 
It may invalidate TLB entries and paging-structure-cache entries associated with PCIDs other than the specified 
PCID. 


MOV to CRO may invalidate TLB entries even if CRO.PG is not changing. For example, this may occur if either 
CRO.CD or CRO.NW is modified. 


MOV to CR3 may invalidate TLB entries for global pages. If CR4.PCIDE = 1 and bit 63 of the instruction’s source 
operand is 0, it may invalidate TLB entries and entries in the paging-structure caches associated with PCIDs 
other than the PCID it is establishing. It may invalidate entries if CR4.PCIDE = 1 and bit 63 of the instruction’s 
source operand is 1. 


MOV to CR4 may invalidate TLB entries when changing CR4.PSE or when changing CR4.SMEP from 1 to 0. 


On a processor supporting Hyper-Threading Technology, invalidations performed on one logical processor may 
invalidate entries in the TLBs and paging-structure caches used by other logical processors. 


(Other instructions and operations may invalidate entries in the TLBs and the paging-structure caches, but the 
instructions identified above are recommended. ) 


In addition to the instructions identified above, page faults invalidate entries in the TLBs and paging-structure 
caches. In particular, a page-fault exception resulting from an attempt to use a linear address will invalidate any 
TLB entries that are for a page number corresponding to that linear address and that are associated with the 
current PCID. It also invalidates all entries in the paging-structure caches that would be used for that linear address 
and that are associated with the current PCID.? These invalidations ensure that the page-fault exception will not 
recur (if the faulting instruction is re-executed) if it would not be caused by the contents of the paging structures in 


If CR4.PGE is changing from 0 to 1, there were no global TLB entries before the execution; if CR4.PGE is changing from 1 to O, there 
will be no global TLB entries after the execution. 


Task switches do not occur in |A-32e mode and thus cannot occur with 4-level paging. Since CR4.PCIDE can be set only with 4-level 
paging, task switches occur only with CR4.PCIDE = 0. 

Unlike INVLPG, page faults need not invalidate all entries in the paging-structure caches, only those that would be used to translate 
the faulting linear address. 
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memory (and if, therefore, it resulted from cached entries that were not invalidated after the paging structures 
were modified in memory). 


As noted in Section 4.10.2, some processors may choose to cache multiple smaller-page TLB entries for a transla- 
tion specified by the paging structures to use a page larger than 4 KBytes. There is no way for software to be aware 
that multiple translations for smaller pages have been used for a large page. The INVLPG instruction and page 
faults provide the same assurances that they provide when a single TLB entry is used: they invalidate all TLB 
entries corresponding to the translation specified by the paging structures. 


4.10.4.2 Recommended Invalidation 
The following items provide some recommendations regarding when software should perform invalidations: 


¢ If software modifies a paging-structure entry that maps a page (rather than referencing another paging 
structure), it should execute INVLPG for any linear address with a page number whose translation uses that 
paging-structure entry.! 


(If the paging-structure entry may be used in the translation of different page numbers — see Section 4.10.3.3 
— software should execute INVLPG for linear addresses with each of those page numbers; alternatively, it could 
use MOV to CR3 or MOV to CR4.) 


¢ If software modifies a paging-structure entry that references another paging structure, it may use one of the 
following approaches depending upon the types and number of translations controlled by the modified entry: 


— Execute INVLPG for linear addresses with each of the page numbers with translations that would use the 
entry. However, if no page numbers that would use the entry have translations (e.g., because the P flags 
are 0 in all entries in the paging structure referenced by the modified entry), it remains necessary to 
execute INVLPG at least once. 


— Execute MOV to CR3 if the modified entry controls no global pages. 
— Execute MOV to CR4 to modify CR4.PGE. 


° If CR4.PCIDE = 1 and software modifies a paging-structure entry that does not map a page or in which the G 
flag (bit 8) is 0, additional steps are required if the entry may be used for PCIDs other than the current one. Any 
one of the following suffices: 


— Execute MOV to CR4 to modify CR4.PGE, either immediately or before again using any of the affected 
PCIDs. For example, software could use different (previously unused) PCIDs for the processes that used the 
affected PCIDs. 


— For each affected PCID, execute MOV to CR3 to make that PCID current (and to load the address of the 
appropriate PML4 table). If the modified entry controls no global pages and bit 63 of the source operand to 
MOV to CR3 was 0, no further steps are required. Otherwise, execute INVLPG for linear addresses with each 
of the page numbers with translations that would use the entry; if no page numbers that would use the 
entry have translations, execute INVLPG at least once. 


¢ If software using PAE paging modifies a PDPTE, it should reload CR3 with the register’s current value to ensure 
that the modified PDPTE is loaded into the corresponding PDPTE register (see Section 4.4.1). 


¢ If the nature of the paging structures is such that a single entry may be used for multiple purposes (see Section 
4.10.3.3), software should perform invalidations for all of these purposes. For example, if a single entry might 
serve as both a PDE and PTE, it may be necessary to execute INVLPG with two (or more) linear addresses, one 
that uses the entry as a PDE and one that uses it as a PTE. (Alternatively, software could use MOV to CR3 or 
MOV to CR4.) 


¢ As noted in Section 4.10.2, the TLBs may subsequently contain multiple translations for the address range if 
software modifies the paging structures so that the page size used for a 4-KByte range of linear addresses 
changes. A reference to a linear address in the address range may use any of these translations. 


Software wishing to prevent this uncertainty should not write to a paging-structure entry in a way that would 
change, for any linear address, both the page size and either the page frame, access rights, or other attributes. 
It can instead use the following algorithm: first clear the P flag in the relevant paging-structure entry (e.g., 


1. One execution of INVLPG is sufficient even for a page with size greater than 4 KBytes. 
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PDE); then invalidate any translations for the affected linear addresses (see above); and then modify the 
relevant paging-structure entry to set the P flag and establish modified translation(s) for the new page size. 


Software should clear bit 63 of the source operand to a MOV to CR3 instruction that establishes a PCID that had 
been used earlier for a different linear-address space (e.g., with a different value in bits 51:12 of CR3). This 
ensures invalidation of any information that may have been cached for the previous linear-address space. 


This assumes that both linear-address spaces use the same global pages and that it is thus not necessary to 
invalidate any global TLB entries. If that is not the case, software should invalidate those entries by executing 
MOV to CR4 to modify CR4.PGE. 


4.10.4.3 Optional Invalidation 


The following items describe cases in which software may choose not to invalidate and the potential consequences 
of that choice: 


If a paging-structure entry is modified to change the P flag from O to 1, no invalidation is necessary. This is 
because no TLB entry or paging-structure cache entry is created with information from a paging-structure entry 
in which the P flag is 0.4 


If a paging-structure entry is modified to change the accessed flag from 0 to 1, no invalidation is necessary 
(assuming that an invalidation was performed the last time the accessed flag was changed from 1 to 0). This is 
because no TLB entry or paging-structure cache entry is created with information from a paging-structure entry 
in which the accessed flag is 0. 


If a paging-structure entry is modified to change the R/W flag from 0 to 1, failure to perform an invalidation 
may result in a “spurious” page-fault exception (e.g., in response to an attempted write access) but no other 
adverse behavior. Such an exception will occur at most once for each affected linear address (see Section 
4.10.4.1). 


If CR4.SMEP = 0 and a paging-structure entry is modified to change the U/S flag from 0 to 1, failure to perform 
an invalidation may result in a “spurious” page-fault exception (e.g., in response to an attempted user-mode 
access) but no other adverse behavior. Such an exception will occur at most once for each affected linear 
address (see Section 4.10.4.1). 


If a paging-structure entry is modified to change the XD flag from 1 to O, failure to perform an invalidation may 
result in a “spurious” page-fault exception (e.g., in response to an attempted instruction fetch) but no other 
adverse behavior. Such an exception will occur at most once for each affected linear address (see Section 
4.10.4.1). 


If a paging-structure entry is modified to change the accessed flag from 1 to O, failure to perform an invali- 
dation may result in the processor not setting that bit in response to a subsequent access to a linear address 
whose translation uses the entry. Software cannot interpret the bit being clear as an indication that such an 
access has not occurred. 


If software modifies a paging-structure entry that identifies the final physical address for a linear address 
(either a PTE or a paging-structure entry in which the PS flag is 1) to change the dirty flag from 1 to 0, failure 
to perform an invalidation may result in the processor not setting that bit in response to a subsequent write to 
a linear address whose translation uses the entry. Software cannot interpret the bit being clear as an indication 
that such a write has not occurred. 


The read of a paging-structure entry in translating an address being used to fetch an instruction may appear to 
execute before an earlier write to that paging-structure entry if there is no serializing instruction between the 
write and the instruction fetch. Note that the invalidating instructions identified in Section 4.10.4.1 are all 
serializing instructions. 


Section 4.10.3.3 describes situations in which a single paging-structure entry may contain information cached 
in multiple entries in the paging-structure caches. Because all entries in these caches are invalidated by any 
execution of INVLPG, it is not necessary to follow the modification of such a paging-structure entry by 
executing INVLPG multiple times solely for the purpose of invalidating these multiple cached entries. (It may be 
necessary to do so to invalidate multiple TLB entries.) 


If it is also the case that no invalidation was performed the last time the P flag was changed from 1 to 0, the processor may use a 
TLB entry or paging-structure cache entry that was created when the P flag had earlier been 1. 
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4.10.4.4 Delayed Invalidation 


Required invalidations may be delayed under some circumstances. Software developers should understand that, 
between the modification of a paging-structure entry and execution of the invalidation instruction recommended in 
Section 4.10.4.2, the processor may use translations based on either the old value or the new value of the paging- 
structure entry. The following items describe some of the potential consequences of delayed invalidation: 


¢ Ifa paging-structure entry is modified to change the P flag from 1 to 0, an access to a linear address whose 
translation is controlled by this entry may or may not cause a page-fault exception. 


¢ If a paging-structure entry is modified to change the R/W flag from 0 to 1, write accesses to linear addresses 
whose translation is controlled by this entry may or may not cause a page-fault exception. 


¢ If a paging-structure entry is modified to change the U/S flag from 0 to 1, user-mode accesses to linear 
addresses whose translation is controlled by this entry may or may not cause a page-fault exception. 


¢ If a paging-structure entry is modified to change the XD flag from 1 to O, instruction fetches from linear 
addresses whose translation is controlled by this entry may or may not cause a page-fault exception. 


As noted in Section 8.1.1, an x87 instruction or an SSE instruction that accesses data larger than a quadword may 
be implemented using multiple memory accesses. If such an instruction stores to memory and invalidation has 
been delayed, some of the accesses may complete (writing to memory) while another causes a page-fault excep- 
tion.! In this case, the effects of the completed accesses may be visible to software even though the overall 
instruction caused a fault. 


In some cases, the consequences of delayed invalidation may not affect software adversely. For example, when 
freeing a portion of the linear-address space (by marking paging-structure entries “not present”), invalidation 
using INVLPG may be delayed if software does not re-allocate that portion of the linear-address space or the 
memory that had been associated with it. However, because of speculative execution (or errant software), there 
may be accesses to the freed portion of the linear-address space before the invalidations occur. In this case, the 
following can happen: 


¢ Reads can occur to the freed portion of the linear-address space. Therefore, invalidation should not be delayed 
for an address range that has read side effects. 


¢ The processor may retain entries in the TLBs and paging-structure caches for an extended period of time. 
Software should not assume that the processor will not use entries associated with a linear address simply 
because time has passed. 


¢ As noted in Section 4.10.3.1, the processor may create an entry in a paging-structure cache even if there are 
no translations for any linear address that might use that entry. Thus, if software has marked “not present” all 
entries in a page table, the processor may subsequently create a PDE-cache entry for the PDE that references 
that page table (assuming that the PDE itself is marked “present”). 


¢ If software attempts to write to the freed portion of the linear-address space, the processor might not generate 
a page fault. (Such an attempt would likely be the result of a software error.) For that reason, the page frames 
previously associated with the freed portion of the linear-address space should not be reallocated for another 
purpose until the appropriate invalidations have been performed. 


4.10.5 Propagation of Paging-Structure Changes to Multiple Processors 


As noted in Section 4.10.4, software that modifies a paging-structure entry may need to invalidate entries in the 
TLBs and paging-structure caches that were derived from the modified entry before it was modified. In a system 
containing more than one logical processor, software must account for the fact that there may be entries in the 
TLBs and paging-structure caches of logical processors other than the one used to modify the paging-structure 
entry. The process of propagating the changes to a paging-structure entry is commonly referred to as “TLB shoot- 
down.” 


TLB shootdown can be done using memory-based semaphores and/or interprocessor interrupts (IPI). The 
following items describe a simple but inefficient example of a TLB shootdown algorithm for processors supporting 
the Intel-64 and IA-32 architectures: 


1. If the accesses are to different pages, this may occur even if invalidation has not been delayed. 
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1. Begin barrier: Stop all but one logical processor; that is, cause all but one to execute the HLT instruction or to 
enter a spin loop. 


Allow the active logical processor to change the necessary paging-structure entries. 


Allow all logical processors to perform invalidations appropriate to the modifications to the paging-structure 
entries. 


4. Allow all logical processors to resume normal operation. 


Alternative, performance-optimized, TLB shootdown algorithms may be developed; however, software developers 
must take care to ensure that the following conditions are met: 


¢ All logical processors that are using the paging structures that are being modified must participate and perform 
appropriate invalidations after the modifications are made. 


° If the modifications to the paging-structure entries are made before the barrier or if there is no barrier, the 
operating system must ensure one of the following: (1) that the affected linear-address range is not used 
between the time of modification and the time of invalidation; or (2) that it is prepared to deal with the conse- 
quences of the affected linear-address range being used during that period. For example, if the operating 
system does not allow pages being freed to be reallocated for another purpose until after the required invalida- 
tions, writes to those pages by errant software will not unexpectedly modify memory that is in use. 


° Software must be prepared to deal with reads, instruction fetches, and prefetch requests to the affected linear- 
address range that are a result of speculative execution that would never actually occur in the executed code 
path. 


When multiple logical processors are using the same linear-address space at the same time, they must coordinate 
before any request to modify the paging-structure entries that control that linear-address space. In these cases, 
the barrier in the TLB shootdown routine may not be required. For example, when freeing a range of linear 
addresses, some other mechanism can assure no logical processor is using that range before the request to free it 
is made. In this case, a logical processor freeing the range can clear the P flags in the PTEs associated with the 
range, free the physical page frames associated with the range, and then signal the other logical processors using 
that linear-address space to perform the necessary invalidations. All the affected logical processors must complete 
their invalidations before the linear-address range and the physical page frames previously associated with that 
range can be reallocated. 


4.11. INTERACTIONS WITH VIRTUAL-MACHINE EXTENSIONS (VMX) 


The architecture for virtual-machine extensions (VMX) includes features that interact with paging. Section 4.11.1 
discusses ways in which VMX-specific control transfers, called VMX transitions specially affect paging. Section 
4.11.2 gives an overview of VMX features specifically designed to support address translation. 


4.11.1 VMX Transitions 


The VMX architecture defines two control transfers called VM entries and VM exits; collectively, these are called 
VMX transitions. VM entries and VM exits are described in detail in Chapter 25 and Chapter 26, respectively, in 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C. The following items identify 
paging-related details: 


° \VMX transitions modify the CRO and CR4 registers and the IA32_EFER MSR concurrently. For this reason, they 
allow transitions between paging modes that would not otherwise be possible: 


— VM entries allow transitions from 4-level paging directly to either 32-bit paging or PAE paging. 


— VM exits allow transitions from either 32-bit paging or PAE paging directly to 4-level paging or 5-level 
paging. 
° VMX transitions that result in PAE paging load the PDPTE registers (see Section 4.4.1) as follows: 


— VM entries load the PDPTE registers either from the physical address being loaded into CR3 or from the 
virtual-machine control structure (VMCS); see Section 25.3.2.4. 


— VM exits load the PDPTE registers from the physical address being loaded into CR3; see Section 26.5.4. 
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¢ VMxX transitions invalidate the TLBs and paging-structure caches based on certain control settings. See Section 
25.3.2.5 and Section 26.5.5 in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
3C. 


4.11.2 | VMX Support for Address Translation 


Chapter 27, “VMX Support for Address Translation,” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 3C describe two features of the virtual-machine extensions (VMX) that interact directly with 
paging. These are virtual-processor identifiers (VPIDs) and the extended page table mechanism (EPT). 


VPIDs provide a way for software to identify to the processor the address spaces for different “virtual processors.” 
The processor may use this identification to maintain concurrently information for multiple address spaces in its 
TLBs and paging-structure caches, even when non-zero PCIDs are not being used. See Section 27.1 for details. 


When EPT is in use, the addresses in the paging-structures are not used as physical addresses to access memory 
and memory-mapped I/O. Instead, they are treated as guest-physical addresses and are translated through a 

set of EPT paging structures to produce physical addresses. EPT can also specify its own access rights and memory 
typing; these are used on conjunction with those specified in this chapter. See Section 27.2 for more information. 


Both VPIDs and EPT may change the way that a processor maintains information in TLBs and paging structure 
caches and the ways in which software can manage that information. Some of the behaviors documented in 
Section 4.10 may change. See Section 27.3 for details. 


4.12 USING PAGING FOR VIRTUAL MEMORY 


With paging, portions of the linear-address space need not be mapped to the physical-address space; data for the 
unmapped addresses can be stored externally (e.g., on disk). This method of mapping the linear-address space is 
referred to as virtual memory or demand-paged virtual memory. 


Paging divides the linear address space into fixed-size pages that can be mapped into the physical-address space 
and/or external storage. When a program (or task) references a linear address, the processor uses paging to trans- 
late the linear address into a corresponding physical address if such an address is defined. 


If the page containing the linear address is not currently mapped into the physical-address space, the processor 
generates a page-fault exception as described in Section 4.7. The handler for page-fault exceptions typically 
directs the operating system or executive to load data for the unmapped page from external storage into physical 
memory (perhaps writing a different page from physical memory out to external storage in the process) and to 
map it using paging (by updating the paging structures). When the page has been loaded into physical memory, a 
return from the exception handler causes the instruction that generated the exception to be restarted. 


Paging differs from segmentation through its use of fixed-size pages. Unlike segments, which usually are the same 
size as the code or data structures they hold, pages have a fixed size. If segmentation is the only form of address 
translation used, a data structure present in physical memory will have all of its parts in memory. If paging is used, 
a data structure can be partly in memory and partly in disk storage. 


4.13, MAPPING SEGMENTS TO PAGES 


The segmentation and paging mechanisms provide support for a wide variety of approaches to memory manage- 
ment. When segmentation and paging are combined, segments can be mapped to pages in several ways. To imple- 
ment a flat (unsegmented) addressing environment, for example, all the code, data, and stack modules can be 
mapped to one or more large segments (up to 4-GBytes) that share same range of linear addresses (see 

Figure 3-2 in Section 3.2.2). Here, segments are essentially invisible to applications and the operating-system or 
executive. If paging is used, the paging mechanism can map a single linear-address space (contained in a single 
segment) into virtual memory. Alternatively, each program (or task) can have its own large linear-address space 
(contained in its own segment), which is mapped into virtual memory through its own paging structures. 


Segments can be smaller than the size of a page. If one of these segments is placed in a page which is not shared 
with another segment, the extra memory is wasted. For example, a small data structure, such as a 1-Byte sema- 
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phore, occupies 4 KBytes if it is placed in a page by itself. If many semaphores are used, it is more efficient to pack 
them into a single page. 


The Intel-64 and IA-32 architectures do not enforce correspondence between the boundaries of pages and 
segments. A page can contain the end of one segment and the beginning of another. Similarly, a segment can 
contain the end of one page and the beginning of another. 


Memory-management software may be simpler and more efficient if it enforces some alignment between page and 
segment boundaries. For example, if a segment which can fit in one page is placed in two pages, there may be 
twice as much paging overhead to support access to that segment. 


One approach to combining paging and segmentation that simplifies memory-management software is to give each 
segment its own page table, as shown in Figure 4-13. This convention gives the segment a single entry in the page 
directory, and this entry provides the access control information for paging the entire segment. 


Page Frames 


LDT Page Directory Page Tables 
PTE — 
PTE > 
>| PTE IJ 
Seg. Descript.-—> PDE 
Seg. Descript.;-—> PDE = 
PTE i 
|_| PTE tJ 


Figure 4-13. Memory Management Convention That Assigns a Page Table 
to Each Segment 
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In protected mode, the Intel 64 and IA-32 architectures provide a protection mechanism that operates at both the 
segment level and the page level. This protection mechanism provides the ability to limit access to certain 
segments or pages based on privilege levels (four privilege levels for segments and two privilege levels for pages). 
For example, critical operating-system code and data can be protected by placing them in more privileged 
segments than those that contain applications code. The processor's protection mechanism will then prevent appli- 
cation code from accessing the operating-system code and data in any but a controlled, defined manner. 


Segment and page protection can be used at all stages of software development to assist in localizing and detecting 
design problems and bugs. It can also be incorporated into end-products to offer added robustness to operating 
systems, utilities software, and applications software. 


When the protection mechanism is used, each memory reference is checked to verify that it satisfies various 
protection checks. All checks are made before the memory cycle is started; any violation results in an exception. 
Because checks are performed in parallel with address translation, there is no performance penalty. The protection 
checks that are performed fall into the following categories: 


¢ Limit checks. 

° Type checks. 

° Privilege level checks. 

°* Restriction of addressable domain. 

° Restriction of procedure entry-points. 
¢ Restriction of instruction set. 


All protection violation results in an exception being generated. See Chapter 6, “Interrupt and Exception Handling,” 
for an explanation of the exception mechanism. This chapter describes the protection mechanism and the viola- 
tions which lead to exceptions. 


The following sections describe the protection mechanism available in protected mode. See Chapter 19, “8086 
Emulation,” for information on protection in real-address and virtual-8086 mode. 


5.1 ENABLING AND DISABLING SEGMENT AND PAGE PROTECTION 


Setting the PE flag in register CRO causes the processor to switch to protected mode, which in turn enables the 
segment-protection mechanism. Once in protected mode, there is no control bit for turning the protection mecha- 
nism on or off. The part of the segment-protection mechanism that is based on privilege levels can essentially be 
disabled while still in protected mode by assigning a privilege level of 0 (most privileged) to all segment selectors 
and segment descriptors. This action disables the privilege level protection barriers between segments, but other 
protection checks such as limit checking and type checking are still carried out. 


Page-level protection is automatically enabled when paging is enabled (by setting the PG flag in register CRO). Here 
again there is no mode bit for turning off page-level protection once paging is enabled. However, page-level protec- 
tion can be disabled by performing the following operations: 


¢ Clear the WP flag in control register CRO. 
° Set the read/write (R/W) and user/supervisor (U/S) flags for each page-directory and page-table entry. 
This action makes each page a writable, user page, which in effect disables page-level protection. 
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5.2 FIELDS AND FLAGS USED FOR SEGMENT-LEVEL AND 


PAGE-LEVEL PROTECTION 


The processor’s protection mechanism uses the following fields and flags in the system data structures to control 
access to segments and pages: 


Descriptor type (S) flag — (Bit 12 in the second doubleword of a segment descriptor.) Determines if the 
segment descriptor is for a system segment or a code or data segment. 


Type field — (Bits 8 through 11 in the second doubleword of a segment descriptor.) Determines the type of 
code, data, or system segment. 


Limit field — (Bits 0 through 15 of the first doubleword and bits 16 through 19 of the second doubleword of a 
segment descriptor.) Determines the size of the segment, along with the G flag and E flag (for data segments). 


G flag — (Bit 23 in the second doubleword of a segment descriptor. ) Determines the size of the segment, along 
with the limit field and E flag (for data segments). 


E flag — (Bit 10 in the second doubleword of a data-segment descriptor.) Determines the size of the segment, 
along with the limit field and G flag. 


Descriptor privilege level (DPL) field — (Bits 13 and 14 in the second doubleword of a segment descriptor. ) 
Determines the privilege level of the segment. 


Requested privilege level (RPL) field — (Bits 0 and 1 of any segment selector.) Specifies the requested 
privilege level of a segment selector. 


Current privilege level (CPL) field — (Bits 0 and 1 of the CS segment register.) Indicates the privilege level 
of the currently executing program or procedure. The term current privilege level (CPL) refers to the setting of 
this field. 


User/supervisor (U/S) flag — (Bit 2 of paging-structure entries.) Determines the type of page: user or 
supervisor. 


Read/write (R/W) flag — (Bit 1 of paging-structure entries.) Determines the type of access allowed to a 
page: read-only or read/write. 


Execute-disable (XD) flag — (Bit 63 of certain paging-structure entries.) Determines the type of access 
allowed to a page: executable or not-executable. 


Figure 5-1 shows the location of the various fields and flags in the data-, code-, and system-segment descriptors; 
Figure 3-6 shows the location of the RPL (or CPL) field in a segment selector (or the CS register); and Chapter 4 
identifies the locations of the U/S, R/W, and XD flags in the paging-structure entries. 
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Many different styles of protection schemes can be implemented with these fields and flags. When the operating 
system creates a descriptor, it places values in these fields and flags in keeping with the particular protection style 
chosen for an operating system or executive. Application programs do not generally access or modify these fields 


and flags. 


The following sections describe how the processor uses these fields and flags to perform the various categories of 
checks described in the introduction to this chapter. 


5.2.1 


Code-Segment Descriptor in 64-bit Mode 


Code segments continue to exist in 64-bit mode even though, for address calculations, the segment base is treated 
as zero. Some code-segment (CS) descriptor content (the base address and limit fields) is ignored; the remaining 
fields function normally (except for the readable bit in the type field). 


Code segment descriptors and selectors are needed in IA-32e mode to establish the processor’s operating mode 
and execution privilege-level. The usage is as follows: 
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° IA-32e mode uses a previously unused bit in the CS descriptor. Bit 53 is defined as the 64-bit (L) flag and is 
used to select between 64-bit mode and compatibility mode when IA-32e mode is active (IA32_EFER.LMA = 1). 
See Figure 5-2. 


— If CS.L = 0 and IA-32e mode is active, the processor is running in compatibility mode. In this case, CS.D 
selects the default size for data and addresses. If CS.D = 0, the default data and address size is 16 bits. If 
CS.D = 1, the default data and address size is 32 bits. 


— If CS.L = 1 and IA-32e mode is active, the only valid setting is CS.D = 0. This setting indicates a default 
operand size of 32 bits and a default address size of 64 bits. The CS.L = 1 and CS.D = 1 bit combination is 
reserved for future use and a #GP fault will be generated on an attempt to use a code segment with these 
bits set in IA-32e mode. 


¢ In IA-32e mode, the CS descriptor’s DPL is used for execution privilege checks (as in legacy 32-bit mode). 
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Figure 5-2. Descriptor Fields with Flags used in |A-32e Mode 


5.3 LIMIT CHECKING 


The limit field of a segment descriptor prevents programs or procedures from addressing memory locations outside 
the segment. The effective value of the limit depends on the setting of the G (granularity) flag (see Figure 5-1). For 
data segments, the limit also depends on the E (expansion direction) flag and the B (default stack pointer size 
and/or upper bound) flag. The E flag is one of the bits in the type field when the segment descriptor is for a data- 
segment type. 


When the G flag is clear (byte granularity), the effective limit is the value of the 20-bit limit field in the segment 
descriptor. Here, the limit ranges from 0 to FFFFFH (1 MByte). When the G flag is set (4-KByte page granularity), 
the processor scales the value in the limit field by a factor of 2 (4 KBytes). In this case, the effective limit ranges 
from FFFH (4 KBytes) to FFFFFFFFH (4 GBytes). Note that when scaling is used (G flag is set), the lower 12 bits of 
a segment offset (address) are not checked against the limit; for example, note that if the segment limit is 0, 
offsets 0 through FFFH are still valid. 


For all types of segments except expand-down data segments, the effective limit is the last address that is allowed 
to be accessed in the segment, which is one less than the size, in bytes, of the segment. The processor causes a 
general-protection exception (or, if the segment is SS, a stack-fault exception) any time an attempt is made to 
access the following addresses in a segment: 


¢ A byte at an offset greater than the effective limit 

¢ Aword at an offset greater than the (effective-limit - 1) 

¢ A doubleword at an offset greater than the (effective-limit - 3) 
¢ A quadword at an offset greater than the (effective-limit - 7) 
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¢ A double quadword at an offset greater than the (effective limit - 15) 


When the effective limit is FFFFFFFFH (4 GBytes), these accesses may or may not cause the indicated exceptions. 
Behavior is implementation-specific and may vary from one execution to another. 


For expand-down data segments, the segment limit has the same function but is interpreted differently. Here, the 
effective limit specifies the last address that is not allowed to be accessed within the segment; the range of valid 
offsets is from (effective-limit + 1) to FFFFFFFFH if the B flag is set and from (effective-limit + 1) to FFFFH if the B 
flag is clear. An expand-down segment has maximum size when the segment limit is 0. 


Limit checking catches programming errors such as runaway code, runaway subscripts, and invalid pointer calcu- 
lations. These errors are detected when they occur, so identification of the cause is easier. Without limit checking, 
these errors could overwrite code or data in another segment. 


In addition to checking segment limits, the processor also checks descriptor table limits. The GDTR and IDTR regis- 
ters contain 16-bit limit values that the processor uses to prevent programs from selecting a segment descriptors 
outside the respective descriptor tables. The LDTR and task registers contain 32-bit segment limit value (read from 
the segment descriptors for the current LDT and TSS, respectively). The processor uses these segment limits to 
prevent accesses beyond the bounds of the current LDT and TSS. See Section 3.5.1, “Segment Descriptor Tables,” 
for more information on the GDT and LDT limit fields; see Section 6.10, “Interrupt Descriptor Table (IDT),” for more 
information on the IDT limit field; and see Section 7.2.4, “Task Register,” for more information on the TSS segment 
limit field. 


5.3.1 Limit Checking in 64-bit Mode 


In 64-bit mode, the processor does not perform runtime limit checking on code or data segments. However, the 
processor does check descriptor-table limits. 


5.4 TYPE CHECKING 


Segment descriptors contain type information in two places: 
¢ The S (descriptor type) flag. 
¢ The type field. 


The processor uses this information to detect programming errors that result in an attempt to use a segment or 
gate in an incorrect or unintended manner. 


The S flag indicates whether a descriptor is a system type or a code or data type. The type field provides 4 addi- 
tional bits for use in defining various types of code, data, and system descriptors. Table 3-1 shows the encoding of 
the type field for code and data descriptors; Table 3-2 shows the encoding of the field for system descriptors. 


The processor examines type information at various times while operating on segment selectors and segment 
descriptors. The following list gives examples of typical operations where type checking is performed (this list is not 
exhaustive): 


° When a segment selector is loaded into a segment register — Certain segment registers can contain only 
certain descriptor types, for example: 


— The CS register only can be loaded with a selector for a code segment. 


— Segment selectors for code segments that are not readable or for system segments cannot be loaded into 
data-segment registers (DS, ES, FS, and GS). 


— Only segment selectors of writable data segments can be loaded into the SS register. 
¢ When a segment selector is loaded into the LDTR or task register — For example: 

— The LDTR can only be loaded with a selector for an LDT. 

— The task register can only be loaded with a segment selector for a TSS. 


° When instructions access segments whose descriptors are already loaded into segment registers — 
Certain segments can be used by instructions only in certain predefined ways, for example: 


— No instruction may write into an executable segment. 
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— No instruction may write into a data segment if it is not writable. 
— No instruction may read an executable segment unless the readable flag is set. 


¢ When an instruction operand contains a segment selector — Certain instructions can access segments 
or gates of only a particular type, for example: 


— A far CALL or far JMP instruction can only access a segment descriptor for a conforming code segment, 
nonconforming code segment, call gate, task gate, or TSS. 


— The LLDT instruction must reference a segment descriptor for an LDT. 
— The LTR instruction must reference a segment descriptor for a TSS. 


— The LAR instruction must reference a segment or gate descriptor for an LDT, TSS, call gate, task gate, code 
segment, or data segment. 


— The LSL instruction must reference a segment descriptor for a LDT, TSS, code segment, or data segment. 
— IDT entries must be interrupt, trap, or task gates. 
* During certain internal operations — For example: 


— Ona far call or far jump (executed with a far CALL or far JMP instruction), the processor determines the 
type of control transfer to be carried out (call or jump to another code segment, a call or jump through a 
gate, or a task switch) by checking the type field in the segment (or gate) descriptor pointed to by the 
segment (or gate) selector given as an operand in the CALL or JMP instruction. If the descriptor type is for 
a code segment or call gate, a call or jump to another code segment is indicated; if the descriptor type is for 
a TSS or task gate, a task switch is indicated. 


— Onacall or jump through a call gate (or on an interrupt- or exception-handler call through a trap or 
interrupt gate), the processor automatically checks that the segment descriptor being pointed to by the 
gate is for a code segment. 


— On acall or jump to a new task through a task gate (or on an interrupt- or exception-handler call to a new 
task through a task gate), the processor automatically checks that the segment descriptor being pointed to 
by the task gate is for a TSS. 


— On acall or jump to a new task by a direct reference to a TSS, the processor automatically checks that the 
segment descriptor being pointed to by the CALL or JMP instruction is for a TSS. 


— Onreturn from a nested task (initiated by an IRET instruction), the processor checks that the previous task 
link field in the current TSS points to a TSS. 


5.4.1 Null Segment Selector Checking 


Attempting to load a null segment selector (see Section 3.4.2, “Segment Selectors”) into the CS or SS segment 
register generates a general-protection exception (#GP). A null segment selector can be loaded into the DS, ES, 
FS, or GS register, but any attempt to access a segment through one of these registers when it is loaded with a null 
segment selector results ina #GP exception being generated. Loading unused data-segment registers with a null 
segment selector is a useful method of detecting accesses to unused segment registers and/or preventing 
unwanted accesses to data segments. 


5.4.1.1 NULL Segment Checking in 64-bit Mode 


In 64-bit mode, the processor does not perform runtime checking on NULL segment selectors. The processor does 
not cause a #GP fault when an attempt is made to access memory where the referenced segment register has a 
NULL segment selector. 


5.5 PRIVILEGE LEVELS 


The processor’s segment-protection mechanism recognizes 4 privilege levels, numbered from 0 to 3. The greater 
numbers mean lesser privileges. Figure 5-3 shows how these levels of privilege can be interpreted as rings of 
protection. 
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The center (reserved for the most privileged code, data, and stacks) is used for the segments containing the critical 
software, usually the kernel of an operating system. Outer rings are used for less critical software. (Systems that 
use only 2 of the 4 possible privilege levels should use levels 0 and 3.) 


Protection Rings 


Operating 
System 
Kernel 


Operating System 
Services A Level 1 


Applications 


Figure 5-3. Protection Rings 


The processor uses privilege levels to prevent a program or task operating at a lesser privilege level from accessing 
a segment with a greater privilege, except under controlled situations. When the processor detects a privilege level 
violation, it generates a general-protection exception (#GP). 


To carry out privilege-level checks between code segments and data segments, the processor recognizes the 
following three types of privilege levels: 


Current privilege level (CPL) — The CPL is the privilege level of the currently executing program or task. It 
is stored in bits O and 1 of the CS and SS segment registers. Normally, the CPL is equal to the privilege level of 
the code segment from which instructions are being fetched. The processor changes the CPL when program 
control is transferred to a code segment with a different privilege level. The CPL is treated slightly differently 
when accessing conforming code segments. Conforming code segments can be accessed from any privilege 
level that is equal to or numerically greater (less privileged) than the DPL of the conforming code segment. 
Also, the CPL is not changed when the processor accesses a conforming code segment that has a different 
privilege level than the CPL. 


Descriptor privilege level (DPL) — The DPLis the privilege level of a segment or gate. It is stored in the DPL 
field of the segment or gate descriptor for the segment or gate. When the currently executing code segment 
attempts to access a segment or gate, the DPL of the segment or gate is compared to the CPL and RPL of the 
segment or gate selector (as described later in this section). The DPL is interpreted differently, depending on 
the type of segment or gate being accessed: 


— Data segment — The DPL indicates the numerically highest privilege level that a program or task can have 
to be allowed to access the segment. For example, if the DPL of a data segment is 1, only programs running 
at a CPL of 0 or 1 can access the segment. 


— Nonconforming code segment (without using a call gate) — The DPL indicates the privilege level that 
a program or task must be at to access the segment. For example, if the DPL of a nonconforming code 
segment is 0, only programs running at a CPL of 0 can access the segment. 


— Call gate — The DPL indicates the numerically highest privilege level that the currently executing program 
or task can be at and still be able to access the call gate. (This is the same access rule as for a data 
segment.) 


— Conforming code segment and nonconforming code segment accessed through a call gate — The 
DPL indicates the numerically lowest privilege level that a program or task can have to be allowed to access 
the segment. For example, if the DPL of a conforming code segment is 2, programs running at a CPL of 0 or 
1 cannot access the segment. 


Vol. 3A 5-7 


PROTECTION 


— TSS — The DPL indicates the numerically highest privilege level that the currently executing program or 
task can be at and still be able to access the TSS. (This is the same access rule as for a data segment. ) 


Requested privilege level (RPL) — The RPL is an override privilege level that is assigned to segment 
selectors. It is stored in bits 0 and 1 of the segment selector. The processor checks the RPL along with the CPL 
to determine if access to a segment is allowed. Even if the program or task requesting access to a segment has 
sufficient privilege to access the segment, access is denied if the RPL is not of sufficient privilege level. That is, 
if the RPL of a segment selector is numerically greater than the CPL, the RPL overrides the CPL, and vice versa. 
The RPL can be used to ensure that privileged code does not access a segment on behalf of an application 
program unless the program itself has access privileges for that segment. See Section 5.10.4, “Checking Caller 
Access Privileges (ARPL Instruction),” for a detailed description of the purpose and typical use of the RPL. 


Privilege levels are checked when the segment selector of a segment descriptor is loaded into a segment register. 
The checks used for data access differ from those used for transfers of program control among code segments; 
therefore, the two kinds of accesses are considered separately in the following sections. 


5.6 PRIVILEGE LEVEL CHECKING WHEN ACCESSING DATA SEGMENTS 


To access operands in a data segment, the segment selector for the data segment must be loaded into the data- 
segment registers (DS, ES, FS, or GS) or into the stack-segment register (SS). (Segment registers can be loaded 
with the MOV, POP, LDS, LES, LFS, LGS, and LSS instructions.) Before the processor loads a segment selector into 
a segment register, it performs a privilege check (see Figure 5-4) by comparing the privilege levels of the currently 
running program or task (the CPL), the RPL of the segment selector, and the DPL of the segment’s segment 
descriptor. The processor loads the segment selector into the segment register if the DPL is numerically greater 
than or equal to both the CPL and the RPL. Otherwise, a general-protection fault is generated and the segment 
register is not loaded. 


CS Register 
CPL 


Segment Selector 
For Data Segment 


RPL 


Data-Segment Descriptor Privilege 
Check 
DPL 


Figure 5-4. Privilege Check for Data Access 


Figure 5-5 shows four procedures (located in codes segments A, B, C, and D), each running at different privilege 
levels and each attempting to access the same data segment. 


1. 


The procedure in code segment A is able to access data segment E using segment selector E1, because the CPL 
of code segment A and the RPL of segment selector E1 are equal to the DPL of data segment E. 


The procedure in code segment B is able to access data segment E using segment selector E2, because the CPL 
of code segment B and the RPL of segment selector E2 are both numerically lower than (more privileged) than 
the DPL of data segment E. A code segment B procedure can also access data segment E using segment 
selector E1. 


The procedure in code segment C is not able to access data segment E using segment selector E3 (dotted line), 
because the CPL of code segment C and the RPL of segment selector E3 are both numerically greater than (less 
privileged) than the DPL of data segment E. Even if a code segment C procedure were to use segment selector 
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E1 or E2, such that the RPL would be acceptable, it still could not access data segment E because its CPL is not 
privileged enough. 


4. The procedure in code segment D should be able to access data segment E because code segment D’s CPL is 
numerically less than the DPL of data segment E. However, the RPL of segment selector E3 (which the code 
segment D procedure is using to access data segment E) is numerically greater than the DPL of data segment 
E, so access is not allowed. If the code segment D procedure were to use segment selector Ei or E2 to access 
the data segment, access would be allowed. 
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Figure 5-5. Examples of Accessing Data Segments From Various Privilege Levels 


As demonstrated in the previous examples, the addressable domain of a program or task varies as its CPL changes. 
When the CPL is 0, data segments at all privilege levels are accessible; when the CPL is 1, only data segments at 
privilege levels 1 through 3 are accessible; when the CPL is 3, only data segments at privilege level 3 are acces- 
sible. 


The RPL of a segment selector can always override the addressable domain of a program or task. When properly 
used, RPLs can prevent problems caused by accidental (or intensional) use of segment selectors for privileged data 
segments by less privileged programs or procedures. 


It is important to note that the RPL of a segment selector for a data segment is under software control. For 
example, an application program running at a CPL of 3 can set the RPL for a data- segment selector to 0. With the 
RPL set to 0, only the CPL checks, not the RPL checks, will provide protection against deliberate, direct attempts to 
violate privilege-level security for the data segment. To prevent these types of privilege-level-check violations, a 
program or procedure can check access privileges whenever it receives a data-segment selector from another 
procedure (see Section 5.10.4, “Checking Caller Access Privileges (ARPL Instruction)”). 


5.6.1 Accessing Data in Code Segments 


In some instances it may be desirable to access data structures that are contained in a code segment. The 
following methods of accessing data in code segments are possible: 


¢ Load a data-segment register with a segment selector for a nonconforming, readable, code segment. 
° Load a data-segment register with a segment selector for a conforming, readable, code segment. 


e Use a code-segment override prefix (CS) to read a readable, code segment whose selector is already loaded in 
the CS register. 


The same rules for accessing data segments apply to method 1. Method 2 is always valid because the privilege 
level of a conforming code segment is effectively the same as the CPL, regardless of its DPL. Method 3 is always 
valid because the DPL of the code segment selected by the CS register is the same as the CPL. 


Vol. 3A 5-9 


PROTECTION 


5.7 PRIVILEGE LEVEL CHECKING WHEN LOADING THE SS REGISTER 


Privilege level checking also occurs when the SS register is loaded with the segment selector for a stack segment. 
Here all privilege levels related to the stack segment must match the CPL; that is, the CPL, the RPL of the stack- 
segment selector, and the DPL of the stack-segment descriptor must be the same. If the RPL and DPL are not equal 
to the CPL, a general-protection exception (#GP) is generated. 


5.8 PRIVILEGE LEVEL CHECKING WHEN TRANSFERRING PROGRAM CONTROL 
BETWEEN CODE SEGMENTS 


To transfer program control from one code segment to another, the segment selector for the destination code 
segment must be loaded into the code-segment register (CS). As part of this loading process, the processor exam- 
ines the segment descriptor for the destination code segment and performs various limit, type, and privilege 
checks. If these checks are successful, the CS register is loaded, program control is transferred to the new code 
segment, and program execution begins at the instruction pointed to by the EIP register. 


Program control transfers are carried out with the JMP, CALL, RET, SYSENTER, SYSEXIT, SYSCALL, SYSRET, INT n, 
and IRET instructions, as well as by the exception and interrupt mechanisms. Exceptions, interrupts, and the IRET 
instruction are special cases discussed in Chapter 6, “Interrupt and Exception Handling.” This chapter discusses 
only the JMP, CALL, RET, SYSENTER, SYSEXIT, SYSCALL, and SYSRET instructions. 

A JMP or CALL instruction can reference another code segment in any of four ways: 

¢ The target operand contains the segment selector for the target code segment. 


¢ The target operand points to a call-gate descriptor, which contains the segment selector for the target code 
segment. 


¢ The target operand points to a TSS, which contains the segment selector for the target code segment. 


° The target operand points to a task gate, which points to a TSS, which in turn contains the segment selector for 
the target code segment. 


The following sections describe first two types of references. See Section 7.3, “Task Switching,” for information on 
transferring program control through a task gate and/or TSS. 


The SYSENTER and SYSEXIT instructions are special instructions for making fast calls to and returns from operating 
system or executive procedures. These instructions are discussed in Section 5.8.7, “Performing Fast Calls to 
System Procedures with the SYSENTER and SYSEXIT Instructions.” 


The SYCALL and SYSRET instructions are special instructions for making fast calls to and returns from operating 
system or executive procedures in 64-bit mode. These instructions are discussed in Section 5.8.8, “Fast System 
Calls in 64-Bit Mode.” 


5.8.1 Direct Calls or Jumps to Code Segments 


The near forms of the JMP, CALL, and RET instructions transfer program control within the current code segment, 
so privilege-level checks are not performed. The far forms of the JMP, CALL, and RET instructions transfer control 
to other code segments, so the processor does perform privilege-level checks. 


When transferring program control to another code segment without going through a call gate, the processor 
examines four kinds of privilege level and type information (see Figure 5-6): 


¢ The CPL. (Here, the CPL is the privilege level of the calling code segment; that is, the code segment that 
contains the procedure that is making the call or jump.) 
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Figure 5-6. Privilege Check for Control Transfer Without Using a Gate 


¢ The DPL of the segment descriptor for the destination code segment that contains the called procedure. 
¢ The RPL of the segment selector of the destination code segment. 


¢ The conforming (C) flag in the segment descriptor for the destination code segment, which determines whether 
the segment is a conforming (C flag is set) or nonconforming (C flag is clear) code segment. See Section 
3.4.5.1, “Code- and Data-Segment Descriptor Types,” for more information about this flag. 


The rules that the processor uses to check the CPL, RPL, and DPL depends on the setting of the C flag, as described 
in the following sections. 


5.8.1.1 Accessing Nonconforming Code Segments 


When accessing nonconforming code segments, the CPL of the calling procedure must be equal to the DPL of the 

destination code segment; otherwise, the processor generates a general-protection exception (#GP). For example 

in Figure 5-7: 

* Code segment C is a nonconforming code segment. A procedure in code segment A can call a procedure in code 
segment C (using segment selector C1) because they are at the same privilege level (CPL of code segment A is 
equal to the DPL of code segment C). 


° A procedure in code segment B cannot call a procedure in code segment C (using segment selector C2 or C1) 
because the two code segments are at different privilege levels. 
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Figure 5-7. Examples of Accessing Conforming and Nonconforming Code Segments From Various Privilege Levels 


The RPL of the segment selector that points to a nonconforming code segment has a limited effect on the privilege 
check. The RPL must be numerically less than or equal to the CPL of the calling procedure for a successful control 
transfer to occur. So, in the example in Figure 5-7, the RPLs of segment selectors Ci and C2 could legally be set to 
0, 1, or 2, but not to 3. 


When the segment selector of a nonconforming code segment is loaded into the CS register, the privilege level field 
is not changed; that is, it remains at the CPL (which is the privilege level of the calling procedure). This is true, even 
if the RPL of the segment selector is different from the CPL. 


5.8.1.2 Accessing Conforming Code Segments 


When accessing conforming code segments, the CPL of the calling procedure may be numerically equal to or 
greater than (less privileged) the DPL of the destination code segment; the processor generates a general-protec- 
tion exception (#GP) only if the CPL is less than the DPL. (The segment selector RPL for the destination code 
segment is not checked if the segment is a conforming code segment. ) 


In the example in Figure 5-7, code segment D is a conforming code segment. Therefore, calling procedures in both 
code segment A and B can access code segment D (using either segment selector Di or D2, respectively), because 
they both have CPLs that are greater than or equal to the DPL of the conforming code segment. For conforming 
code segments, the DPL represents the numerically lowest privilege level that a calling procedure may 
be at to successfully make a call to the code segment. 


(Note that segments selectors D1 and D2 are identical except for their respective RPLs. But since RPLs are not 
checked when accessing conforming code segments, the two segment selectors are essentially interchangeable.) 


When program control is transferred to a conforming code segment, the CPL does not change, even if the DPL of 
the destination code segment is less than the CPL. This situation is the only one where the CPL may be different 
from the DPL of the current code segment. Also, since the CPL does not change, no stack switch occurs. 


Conforming segments are used for code modules such as math libraries and exception handlers, which support 
applications but do not require access to protected system facilities. These modules are part of the operating 
system or executive software, but they can be executed at numerically higher privilege levels (less privileged 
levels). Keeping the CPL at the level of a calling code segment when switching to a conforming code segment 
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prevents an application program from accessing nonconforming code segments while at the privilege level (DPL) of 
a conforming code segment and thus prevents it from accessing more privileged data. 


Most code segments are nonconforming. For these segments, program control can be transferred only to code 
segments at the same level of privilege, unless the transfer is carried out through a call gate, as described in the 
following sections. 


5.8.2 Gate Descriptors 


To provide controlled access to code segments with different privilege levels, the processor provides special set of 
descriptors called gate descriptors. There are four kinds of gate descriptors: 


° Call gates 
° Trap gates 
° Interrupt gates 
° Task gates 


Task gates are used for task switching and are discussed in Chapter 7, “Task Management”. Trap and interrupt 
gates are special kinds of call gates used for calling exception and interrupt handlers. The are described in Chapter 
6, “Interrupt and Exception Handling.” This chapter is concerned only with call gates. 


5.8.3 Call Gates 


Call gates facilitate controlled transfers of program control between different privilege levels. They are typically 
used only in operating systems or executives that use the privilege-level protection mechanism. Call gates are also 
useful for transferring program control between 16-bit and 32-bit code segments, as described in Section 20.4, 
“Transferring Control Among Mixed-Size Code Segments.” 


Figure 5-8 shows the format of a call-gate descriptor. A call-gate descriptor may reside in the GDT or in an LDT, but 
not in the interrupt descriptor table (IDT). It performs six functions: 


¢ It specifies the code segment to be accessed. 
¢ It defines an entry point for a procedure in the specified code segment. 
¢ It specifies the privilege level required for a caller trying to access the procedure. 


31 16 15 141312 11 87 6 54 0 
D Type 
Offset in Segment 31:16 P| P #P 000 Pte 4 
L jo|1 | 1 (0 | 0 
31 16 15 0 
Segment Selector Offset in Segment 15:00 0 


DPL. Descriptor Privilege Level 
P Gate Valid 


Figure 5-8. Call-Gate Descriptor 


° If a stack switch occurs, it specifies the number of optional parameters to be copied between stacks. 


¢ It defines the size of values to be pushed onto the target stack: 16-bit gates force 16-bit pushes and 32-bit 
gates force 32-bit pushes. 


¢ It specifies whether the call-gate descriptor is valid. 


The segment selector field in a call gate specifies the code segment to be accessed. The offset field specifies the 
entry point in the code segment. This entry point is generally to the first instruction of a specific procedure. The DPL 
field indicates the privilege level of the call gate, which in turn is the privilege level required to access the selected 
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procedure through the gate. The P flag indicates whether the call-gate descriptor is valid. (The presence of the code 
segment to which the gate points is indicated by the P flag in the code segment’s descriptor.) The parameter count 
field indicates the number of parameters to copy from the calling procedures stack to the new stack if a stack 
switch occurs (see Section 5.8.5, “Stack Switching”). The parameter count specifies the number of words for 16- 
bit call gates and doublewords for 32-bit call gates. 


Note that the P flag in a gate descriptor is normally always set to 1. If it is set to 0, a not present (#NP) exception 
is generated when a program attempts to access the descriptor. The operating system can use the P flag for special 
purposes. For example, it could be used to track the number of times the gate is used. Here, the P flag is initially 

set to 0 causing a trap to the not-present exception handler. The exception handler then increments a counter and 
sets the P flag to 1, so that on returning from the handler, the gate descriptor will be valid. 


5.8.3.1 IA-32e Mode Call Gates 


Call-gate descriptors in 32-bit mode provide a 32-bit offset for the instruction pointer (EIP); 64-bit extensions 
double the size of 32-bit mode call gates in order to store 64-bit instruction pointers (RIP). See Figure 5-9: 


¢ The first eight bytes (bytes 7:0) of a 64-bit mode call gate are similar but not identical to legacy 32-bit mode 
call gates. The parameter-copy-count field has been removed. 


¢ Bytes 11:8 hold the upper 32 bits of the target-segment offset in canonical form. A general-protection 
exception (#GP) is generated if software attempts to use a call gate with a target offset that is not in canonical 
form. 


¢ 16-byte descriptors may reside in the same descriptor table with 16-bit and 32-bit descriptors. A type field, 
used for consistency checking, is defined in bits 12:8 of the 64-bit descriptor’s highest dword (cleared to zero). 
A general-protection exception (#GP) results if an attempt is made to access the upper half of a 64-bit mode 
descriptor as a 32-bit mode descriptor. 
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Figure 5-9. Call-Gate Descriptor in I[A-32e Mode 


° Target code segments referenced by a 64-bit call gate must be 64-bit code segments (CS.L = 1, CS.D = 0). If 
not, the reference generates a general-protection exception, #GP (CS selector). 

* Only 64-bit mode call gates can be referenced in IA-32e mode (64-bit mode and compatibility mode). The 
legacy 32-bit mode call gate type (OCH) is redefined in IA-32e mode as a 64-bit call-gate type; no 32-bit call- 
gate type exists in IA-32e mode. 
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¢ Ifa far call references a 16-bit call gate type (04H) in IA-32e mode, a general-protection exception (#GP) is 
generated. 


When a call references a 64-bit mode call gate, actions taken are identical to those taken in 32-bit mode, with the 
following exceptions: 


¢ Stack pushes are made in eight-byte increments. 
¢ A 64-bit RIP is pushed onto the stack. 
¢ Parameter copying is not performed. 


Use a matching far-return instruction size for correct operation (returns from 64-bit calls must be performed with 
a 64-bit operand-size return to process the stack correctly). 


5.8.4 Accessing a Code Segment Through a Call Gate 


To access a call gate, a far pointer to the gate is provided as a target operand in a CALL or JMP instruction. The 
segment selector from this pointer identifies the call gate (see Figure 5-10); the offset from the pointer is required, 
but not used or checked by the processor. (The offset can be set to any value.) 


When the processor has accessed the call gate, it uses the segment selector from the call gate to locate the 
segment descriptor for the destination code segment. (This segment descriptor can be in the GDT or the LDT.) It 
then combines the base address from the code-segment descriptor with the offset from the call gate to form the 
linear address of the procedure entry point in the code segment. 


As shown in Figure 5-11, four different privilege levels are used to check the validity of a program control transfer 
through a call gate: 


¢ The CPL (current privilege level). 

¢ The RPL (requestor's privilege level) of the call gate’s selector. 

¢ The DPL (descriptor privilege level) of the call gate descriptor. 

¢ The DPL of the segment descriptor of the destination code segment. 

The C flag (conforming) in the segment descriptor for the destination code segment is also checked. 
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Figure 5-10. Call-Gate Mechanism 
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Figure 5-11. Privilege Check for Control Transfer with Call Gate 


The privilege checking rules are different depending on whether the control transfer was initiated with a CALL ora 
JMP instruction, as shown in Table 5-1. 
Table 5-1. Privilege Check Rules for Call Gates 
Instruction Privilege Check Rules 
CALL CPL < call gate DPL; RPL < call gate DPL 


Destination conforming code segment DPL < CPL 


Destination nonconforming code segment DPL < CPL 
JMP CPL < call gate DPL; RPL < call gate DPL 


Destination conforming code segment DPL < CPL 


Destination nonconforming code segment DPL = CPL 


The DPL field of the call-gate descriptor specifies the numerically highest privilege level from which a calling proce- 
dure can access the call gate; that is, to access a call gate, the CPL of a calling procedure must be equal to or less 
than the DPL of the call gate. For example, in Figure 5-15, call gate A has a DPL of 3. So calling procedures at all 
CPLs (0 through 3) can access this call gate, which includes calling procedures in code segments A, B, and C. Call 
gate B has a DPL of 2, so only calling procedures at a CPL or O, 1, or 2 can access call gate B, which includes calling 
procedures in code segments B and C. The dotted line shows that a calling procedure in code segment A cannot 
access Call gate B. 


The RPL of the segment selector to a call gate must satisfy the same test as the CPL of the calling procedure; that 
is, the RPL must be less than or equal to the DPL of the call gate. In the example in Figure 5-15, a calling procedure 
in code segment C can access call gate B using gate selector B2 or B1, but it could not use gate selector B3 to 
access Call gate B. 


If the privilege checks between the calling procedure and call gate are successful, the processor then checks the 
DPL of the code-segment descriptor against the CPL of the calling procedure. Here, the privilege check rules vary 
between CALL and JMP instructions. Only CALL instructions can use call gates to transfer program control to more 
privileged (numerically lower privilege level) nonconforming code segments; that is, to nonconforming code 
segments with a DPL less than the CPL. A JMP instruction can use a call gate only to transfer program control to a 
nonconforming code segment with a DPL equal to the CPL. CALL and JMP instruction can both transfer program 
control to a more privileged conforming code segment; that is, to a conforming code segment with a DPL less than 
or equal to the CPL. 
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If a call is made to a more privileged (numerically lower privilege level) nonconforming destination code segment, 
the CPL is lowered to the DPL of the destination code segment and a stack switch occurs (see Section 5.8.5, “Stack 
Switching”). If a call or jump is made to a more privileged conforming destination code segment, the CPL is not 
changed and no stack switch occurs. 
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Figure 5-12. Example of Accessing Call Gates At Various Privilege Levels 


Call gates allow a single code segment to have procedures that can be accessed at different privilege levels. For 
example, an operating system located in a code segment may have some services which are intended to be used 
by both the operating system and application software (such as procedures for handling character I/O). Call gates 
for these procedures can be set up that allow access at all privilege levels (0 through 3). More privileged call gates 
(with DPLs of 0 or 1) can then be set up for other operating system services that are intended to be used only by 
the operating system (such as procedures that initialize device drivers). 


5.8.5 Stack Switching 


Whenever a call gate is used to transfer program control to a more privileged nonconforming code segment (that 
is, when the DPL of the nonconforming destination code segment is less than the CPL), the processor automatically 
switches to the stack for the destination code segment’s privilege level. This stack switching is carried out to 
prevent more privileged procedures from crashing due to insufficient stack space. It also prevents less privileged 
procedures from interfering (by accident or intent) with more privileged procedures through a shared stack. 


Each task must define up to 4 stacks: one for applications code (running at privilege level 3) and one for each of 

the privilege levels 2, 1, and 0 that are used. (If only two privilege levels are used [3 and O], then only two stacks 
must be defined.) Each of these stacks is located in a separate segment and is identified with a segment selector 
and an offset into the stack segment (a stack pointer). 


The segment selector and stack pointer for the privilege level 3 stack is located in the SS and ESP registers, respec- 
tively, when privilege-level-3 code is being executed and is automatically stored on the called procedure’s stack 
when a stack switch occurs. 


Pointers to the privilege level 0, 1, and 2 stacks are stored in the TSS for the currently running task (see 

Figure 7-2). Each of these pointers consists of a segment selector and a stack pointer (loaded into the ESP 
register). These initial pointers are strictly read-only values. The processor does not change them while the task is 
running. They are used only to create new stacks when calls are made to more privileged levels (numerically lower 


Vol. 3A 5-17 


PROTECTION 


privilege levels). These stacks are disposed of when a return is made from the called procedure. The next time the 
procedure is called, a new stack is created using the initial stack pointer. (The TSS does not specify a stack for priv- 
ilege level 3 because the processor does not allow a transfer of program control from a procedure running at a CPL 
of 0, 1, or 2 to a procedure running at a CPL of 3, except on a return.) 


The operating system is responsible for creating stacks and stack-segment descriptors for all the privilege levels to 
be used and for loading initial pointers for these stacks into the TSS. Each stack must be read/write accessible (as 
specified in the type field of its segment descriptor) and must contain enough space (as specified in the limit field) 
to hold the following items: 


¢ The contents of the SS, ESP, CS, and EIP registers for the calling procedure. 
¢ The parameters and temporary variables required by the called procedure. 
¢ The EFLAGS register and error code, when implicit calls are made to an exception or interrupt handler. 


The stack will need to require enough space to contain many frames of these items, because procedures often call 
other procedures, and an operating system may support nesting of multiple interrupts. Each stack should be large 
enough to allow for the worst case nesting scenario at its privilege level. 


(If the operating system does not use the processor’s multitasking mechanism, it still must create at least one TSS 
for this stack-related purpose. ) 


When a procedure call through a call gate results in a change in privilege level, the processor performs the following 
steps to switch stacks and begin execution of the called procedure at a new privilege level: 


1. Uses the DPL of the destination code segment (the new CPL) to select a pointer to the new stack (segment 
selector and stack pointer) from the TSS. 


2. Reads the segment selector and stack pointer for the stack to be switched to from the current TSS. Any limit 
violations detected while reading the stack-segment selector, stack pointer, or stack-segment descriptor cause 
an invalid TSS (#TS) exception to be generated. 


3. Checks the stack-segment descriptor for the proper privileges and type and generates an invalid TSS (#TS) 
exception if violations are detected. 


4. Temporarily saves the current values of the SS and ESP registers. 
Loads the segment selector and stack pointer for the new stack in the SS and ESP registers. 


Pushes the temporarily saved values for the SS and ESP registers (for the calling procedure) onto the new stack 
(see Figure 5-13). 


7. Copies the number of parameter specified in the parameter count field of the call gate from the calling 
procedure’s stack to the new stack. If the count is 0, no parameters are copied. 


Pushes the return instruction pointer (the current contents of the CS and EIP registers) onto the new stack. 


Loads the segment selector for the new code segment and the new instruction pointer from the call gate into 
the CS and EIP registers, respectively, and begins execution of the called procedure. 


See the description of the CALL instruction in Chapter 3, Instruction Set Reference, in the IA-32 Intel Architecture 
Software Developer’s Manual, Volume 2, for a detailed description of the privilege level checks and other protection 
checks that the processor performs on a far call through a call gate. 
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Figure 5-13. Stack Switching During an Interprivilege-Level Call 


The parameter count field in a call gate specifies the number of data items (up to 31) that the processor should 
copy from the calling procedure’s stack to the stack of the called procedure. If more than 31 data items need to be 
passed to the called procedure, one of the parameters can be a pointer to a data structure, or the saved contents 
of the SS and ESP registers may be used to access parameters in the old stack space. The size of the data items 
passed to the called procedure depends on the call gate size, as described in Section 5.8.3, “Call Gates.” 


5.8.5.1 Stack Switching in 64-bit Mode 


Although protection-check rules for call gates are unchanged from 32-bit mode, stack-switch changes in 64-bit 
mode are different. 


When stacks are switched as part of a 64-bit mode privilege-level change through a call gate, a new SS (stack 
segment) descriptor is not loaded; 64-bit mode only loads an inner-level RSP from the TSS. The new SS is forced 
to NULL and the SS selector’s RPL field is forced to the new CPL. The new SS is set to NULL in order to handle 
nested far transfers (far CALL, INTn, interrupts and exceptions). The old SS and RSP are saved on the new stack. 


On a subsequent far RET, the old SS is popped from the stack and loaded into the SS register. See Table 5-2. 


Table 5-2. 64-Bit-Mode Stack Layout After Far CALL with CPL Change 


32-bit Mode IA-32e mode 
Old SS Selector +12 +24 Old SS Selector 
Old ESP +8 +16 Old RSP 
CS Selector +4 +8 Old CS Selector 
EIP 0 ESP RSP 0 RIP 
< 4Bytes > < 8 Bytes > 


In 64-bit mode, stack operations resulting from a privilege-level-changing far call or far return are eight-bytes wide 
and change the RSP by eight. The mode does not support the automatic parameter-copy feature found in 32-bit 
mode. The call-gate count field is ignored. Software can access the old stack, if necessary, by referencing the old 
stack-segment selector and stack pointer saved on the new process stack. 


In 64-bit mode, far RET is allowed to load a NULL SS under certain conditions. If the target mode is 64-bit mode 
and the target CPL + 3, IRET allows SS to be loaded with a NULL selector. If the called procedure itself is inter- 
rupted, the NULL SS is pushed on the stack frame. On the subsequent far RET, the NULL SS on the stack acts as a 
flag to tell the processor not to load a new SS descriptor. 
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5.8.6 Returning from a Called Procedure 


The RET instruction can be used to perform a near return, a far return at the same privilege level, and a far return 
to a different privilege level. This instruction is intended to execute returns from procedures that were called with 
a CALL instruction. It does not support returns from a JMP instruction, because the JMP instruction does not save a 
return instruction pointer on the stack. 


A near return only transfers program control within the current code segment; therefore, the processor performs 
only a limit check. When the processor pops the return instruction pointer from the stack into the EIP register, it 
checks that the pointer does not exceed the limit of the current code segment. 


On a far return at the same privilege level, the processor pops both a segment selector for the code segment being 
returned to and a return instruction pointer from the stack. Under normal conditions, these pointers should be 
valid, because they were pushed on the stack by the CALL instruction. However, the processor performs privilege 
checks to detect situations where the current procedure might have altered the pointer or failed to maintain the 
stack properly. 


A far return that requires a privilege-level change is only allowed when returning to a less privileged level (that is, 
the DPL of the return code segment is numerically greater than the CPL). The processor uses the RPL field from the 
CS register value saved for the calling procedure (see Figure 5-13) to determine if a return to a numerically higher 
privilege level is required. If the RPL is numerically greater (less privileged) than the CPL, a return across privilege 
levels occurs. 


The processor performs the following steps when performing a far return to a calling procedure (see Figures 6-2 
and 6-4 in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, for an illustration of the 
stack contents prior to and after a return): 


1. Checks the RPL field of the saved CS register value to determine if a privilege level change is required on the 
return. 


2. Loads the CS and EIP registers with the values on the called procedure’s stack. (Type and privilege level checks 
are performed on the code-segment descriptor and RPL of the code- segment selector.) 


3. (If the RET instruction includes a parameter count operand and the return requires a privilege level change.) 
Adds the parameter count (in bytes obtained from the RET instruction) to the current ESP register value (after 
popping the CS and EIP values), to step past the parameters on the called procedure’s stack. The resulting 
value in the ESP register points to the saved SS and ESP values for the calling procedure’s stack. (Note that the 
byte count in the RET instruction must be chosen to match the parameter count in the call gate that the calling 
procedure referenced when it made the original call multiplied by the size of the parameters.) 


4. (If the return requires a privilege level change.) Loads the SS and ESP registers with the saved SS and ESP 
values and switches back to the calling procedure’s stack. The SS and ESP values for the called procedure’s 
stack are discarded. Any limit violations detected while loading the stack-segment selector or stack pointer 
cause a general-protection exception (#GP) to be generated. The new stack-segment descriptor is also checked 
for type and privilege violations. 


5. (If the RET instruction includes a parameter count operand.) Adds the parameter count (in bytes obtained from 
the RET instruction) to the current ESP register value, to step past the parameters on the calling procedure’s 
stack. The resulting ESP value is not checked against the limit of the stack segment. If the ESP value is beyond 
the limit, that fact is not recognized until the next stack operation. 


6. (If the return requires a privilege level change.) Checks the contents of the DS, ES, FS, and GS segment 
registers. If any of these registers refer to segments whose DPL is less than the new CPL (excluding conforming 
code segments), the segment register is loaded with a null segment selector. 


See the description of the RET instruction in Chapter 4 of the Inte/® 64 and IA-32 Architectures Software Devel- 
oper’s Manual, Volume 2B, for a detailed description of the privilege level checks and other protection checks that 
the processor performs on a far return. 


5.8.7 Performing Fast Calls to System Procedures with the 
SYSENTER and SYSEXIT Instructions 


The SYSENTER and SYSEXIT instructions were introduced into the IA-32 architecture in the Pentium II processors 
for the purpose of providing a fast (low overhead) mechanism for calling operating system or executive procedures. 
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SYSENTER is intended for use by user code running at privilege level 3 to access operating system or executive 
procedures running at privilege level 0. SYSEXIT is intended for use by privilege level 0 operating system or exec- 
utive procedures for fast returns to privilege level 3 user code. SYSENTER can be executed from privilege levels 3, 
2, 1, or 0; SYSEXIT can only be executed from privilege level 0. 


The SYSENTER and SYSEXIT instructions are companion instructions, but they do not constitute a call/return pair. 
This is because SYSENTER does not save any state information for use by SYSEXIT on a return. 


The target instruction and stack pointer for these instructions are not specified through instruction operands. 
Instead, they are specified through parameters entered in MSRs and general-purpose registers. 


For SYSENTER, target fields are generated using the following sources: 

° Target code segment — Reads this from IA32_SYSENTER_CS. 

° Target instruction — Reads this from IA32_SYSENTER_EIP. 

¢ Stack segment — Computed by adding 8 to the value in IA32_SYSENTER_CS. 

¢ Stack pointer — Reads this from the IA32_SYSENTER_ESP. 

For SYSEXIT, target fields are generated using the following sources: 

° Target code segment — Computed by adding 16 to the value in the IA32_SYSENTER_CS. 
° Target instruction — Reads this from EDX. 

¢ Stack segment — Computed by adding 24 to the value in IA32_SYSENTER_CS. 

¢ Stack pointer — Reads this from ECX. 


The SYSENTER and SYSEXIT instructions perform “fast” calls and returns because they force the processor into a 
predefined privilege level 0 state when SYSENTER is executed and into a predefined privilege level 3 state when 
SYSEXIT is executed. By forcing predefined and consistent processor states, the number of privilege checks ordi- 
narily required to perform a far call to another privilege levels are greatly reduced. Also, by predefining the target 
context state in MSRs and general-purpose registers eliminates all memory accesses except when fetching the 
target code. 


Any additional state that needs to be saved to allow a return to the calling procedure must be saved explicitly by 
the calling procedure or be predefined through programming conventions. 


5.8.7.1 SYSENTER and SYSEXIT Instructions in IA-32e Mode 


For Intel 64 processors, the SYSENTER and SYSEXIT instructions are enhanced to allow fast system calls from user 
code running at privilege level 3 (in compatibility mode or 64-bit mode) to 64-bit executive procedures running at 
privilege level 0. IA32_SYSENTER_EIP MSR and IA32_SYSENTER_ESP MSR are expanded to hold 64-bit addresses. 
If IA-32e mode is inactive, only the lower 32-bit addresses stored in these MSRs are used. The WRMSR instruction 
ensures that the addresses stored in these MSRs are canonical. Note that, in 64-bit mode, IA32_SYSENTER_CS 
must not contain a NULL selector. 


When SYSENTER transfers control, the following fields are generated and bits set: 

° Target code segment — Reads non-NULL selector from IA32_SYSENTER_CS. 

° New CS attributes — CS base = 0, CS limit = FFFFFFFFH. 

° Target instruction — Reads 64-bit canonical address from IA32_SYSENTER_EIP. 
¢ Stack segment — Computed by adding 8 to the value from IA32_SYSENTER_CS. 
¢ Stack pointer — Reads 64-bit canonical address from IA32_SYSENTER_ESP. 

° New SS attributes — SS base = 0, SS limit = FFFFFFFFH. 


When the SYSEXIT instruction transfers control to 64-bit mode user code using REX.W, the following fields are 
generated and bits set: 


¢ Target code segment — Computed by adding 32 to the value in IA32_SYSENTER_CS. 
¢ New CS attributes — L-bit = 1 (go to 64-bit mode). 

° Target instruction — Reads 64-bit canonical address in RDX. 

¢ Stack segment — Computed by adding 40 to the value of IA32_SYSENTER_CS. 
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¢ Stack pointer — Update RSP using 64-bit canonical address in RCX. 


When SYSEXIT transfers control to compatibility mode user code when the operand size attribute is 32 bits, the 
following fields are generated and bits set: 


° Target code segment — Computed by adding 16 to the value in IA32_SYSENTER_CS. 
° New CS attributes — L-bit = 0 (go to compatibility mode). 

* Target instruction — Fetch the target instruction from 32-bit address in EDX. 

° Stack segment — Computed by adding 24 to the value in IA32_SYSENTER_CS. 

¢ Stack pointer — Update ESP from 32-bit address in ECX. 


5.8.8 Fast System Calls in 64-Bit Mode 


The SYSCALL and SYSRET instructions are designed for operating systems that use a flat memory model (segmen- 
tation is not used). The instructions, along with SYSENTER and SYSEXIT, are suited for IA-32e mode operation. 
SYSCALL and SYSRET, however, are not supported in compatibility mode (or in protected mode). Use CPUID to 
check if SYSCALL and SYSRET are available (CPUID.80000001H.EDX[bit 11] = 1). 


SYSCALL is intended for use by user code running at privilege level 3 to access operating system or executive 
procedures running at privilege level 0. SYSRET is intended for use by privilege level 0 operating system or execu- 
tive procedures for fast returns to privilege level 3 user code. 


Stack pointers for SYSCALL/SYSRET are not specified through model specific registers. The clearing of bits in 
RFLAGS is programmable rather than fixed. SYSCALL/SYSRET save and restore the RFLAGS register. 


For SYSCALL, the processor saves RFLAGS into R11 and the RIP of the next instruction into RCX; it then gets the 
privilege-level 0 target code segment, instruction pointer, stack segment, and flags as follows: 


° Target code segment — Reads a non-NULL selector from IA32_STAR[47:32]. 


* Target instruction pointer — Reads a 64-bit address from IA32_LSTAR. (The WRMSR instruction ensures 
that the value of the IA32_LSTAR MSR is canonical.) 


¢ Stack segment — Computed by adding 8 to the value in IA32_STAR[47:32]. 


¢ Flags — The processor sets RFLAGS to the logical-AND of its current value with the complement of the value in 
the IA32_FMASK MSR. 


When SYSRET transfers control to 64-bit mode user code using REX.W, the processor gets the privilege level 3 
target code segment, instruction pointer, stack segment, and flags as follows: 


° Target code segment — Reads a non-NULL selector from IA32_STAR[63:48] + 16. 
* Target instruction pointer — Copies the value in RCX into RIP. 

° Stack segment — JA32_STAR[63:48] + 8. 

¢ EFLAGS — Loaded from R11. 


When SYSRET transfers control to 32-bit mode user code using a 32-bit operand size, the processor gets the priv- 
ilege level 3 target code segment, instruction pointer, stack segment, and flags as follows: 


° Target code segment — Reads a non-NULL selector from IA32_STAR[63:48]. 
* Target instruction pointer — Copies the value in ECX into EIP. 

° Stack segment — IA32_STAR[63:48] + 8. 

¢ EFLAGS — Loaded from R11. 


It is the responsibility of the OS to ensure the descriptors in the GDT/LDT correspond to the selectors loaded by 
SYSCALL/SYSRET (consistent with the base, limit, and attribute values forced by the instructions). 


See Figure 5-14 for the layout of IA32_STAR, IA32_LSTAR and IA32_FMASK. 
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Figure 5-14. MSRs Used by SYSCALL and SYSRET 


The SYSCALL instruction does not save the stack pointer, and the SYSRET instruction does not restore it. It is likely 
that the OS system-call handler will change the stack pointer from the user stack to the OS stack. If so, it is the 
responsibility of software first to save the user stack pointer. This might be done by user code, prior to executing 
SYSCALL, or by the OS system-call handler after SYSCALL. 


Because the SYSRET instruction does not modify the stack pointer, it is necessary for software to switch back to the 
user stack. The OS may load the user stack pointer (if it was saved after SYSCALL) before executing SYSRET; alter- 
natively, user code may load the stack pointer (if it was saved before SYSCALL) after receiving control from 
SYSRET. 


If the OS loads the stack pointer before executing SYSRET, it must ensure that the handler of any interrupt or 
exception delivered between restoring the stack pointer and successful execution of SYSRET is not invoked with the 
user stack. It can do so using approaches such as the following: 


¢ External interrupts. The OS can prevent an external interrupt from being delivered by clearing EFLAGS.IF 
before loading the user stack pointer. 


¢ Nonmaskable interrupts (NMIs). The OS can ensure that the NMI handler is invoked with the correct stack by 
using the interrupt stack table (IST) mechanism for gate 2 (NMI) in the IDT (see Section 6.14.5, “Interrupt 
Stack Table”). 


° General-protection exceptions (#GP). The SYSRET instruction generates #GP(0) if the value of RCX is not 
canonical. The OS can address this possibility using one or more of the following approaches: 


— Confirming that the value of RCX is canonical before executing SYSRET. 
— Using paging to ensure that the SYSCALL instruction will never save a non-canonical value into RCX. 
— Using the IST mechanism for gate 13 (#GP) in the IDT. 


= PRIVILEGED INSTRUCTIONS 


Some of the system instructions (called “privileged instructions”) are protected from use by application programs. 
The privileged instructions control system functions (such as the loading of system registers). They can be 

executed only when the CPL is 0 (most privileged). If one of these instructions is executed when the CPL is not 0, 
a general-protection exception (#GP) is generated. The following system instructions are privileged instructions: 


¢ LGDT — Load GDT register. 
¢ LLDT — Load LDT register. 
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¢ LTR — Load task register. 

¢ LIDT — Load IDT register. 

¢ MOV (control registers) — Load and store control registers. 
¢ LMSW — Load machine status word. 

* CLTS — Clear task-switched flag in register CRO. 

¢ MOV (debug registers) — Load and store debug registers. 
¢ —INVD — Invalidate cache, without writeback. 

¢ WBINVD — Invalidate cache, with writeback. 

¢ INVLPG —Invalidate TLB entry. 

¢ HLT— Halt processor. 

® RDMSR — Read Model-Specific Registers. 

¢ WRMSR —Write Model-Specific Registers. 

* RDPMC — Read Performance-Monitoring Counter. 

¢ RDTSC — Read Time-Stamp Counter. 


Some of the privileged instructions are available only in the more recent families of Intel 64 and IA-32 processors 
(see Section 21.13, “New Instructions In the Pentium and Later IA-32 Processors”). 


The PCE and TSD flags in register CR4 (bits 4 and 2, respectively) enable the RDPMC and RDTSC instructions, 
respectively, to be executed at any CPL. 


5.10 POINTER VALIDATION 


When operating in protected mode, the processor validates all pointers to enforce protection between segments 
and maintain isolation between privilege levels. Pointer validation consists of the following checks: 


Checking access rights to determine if the segment type is compatible with its use. 
Checking read/write rights. 

Checking if the pointer offset exceeds the segment limit. 

Checking if the supplier of the pointer is allowed to access the segment. 


OTe Gs Nee 


Checking the offset alignment. 


The processor automatically performs first, second, and third checks during instruction execution. Software must 
explicitly request the fourth check by issuing an ARPL instruction. The fifth check (offset alignment) is performed 
automatically at privilege level 3 if alignment checking is turned on. Offset alignment does not affect isolation of 

privilege levels. 


5.10.1 Checking Access Rights (LAR Instruction) 


When the processor accesses a segment using a far pointer, it performs an access rights check on the segment 
descriptor pointed to by the far pointer. This check is performed to determine if type and privilege level (DPL) of the 
segment descriptor are compatible with the operation to be performed. For example, when making a far call in 
protected mode, the segment-descriptor type must be for a conforming or nonconforming code segment, a call 
gate, a task gate, ora TSS. Then, if the call is to a nonconforming code segment, the DPL of the code segment must 
be equal to the CPL, and the RPL of the code segment’s segment selector must be less than or equal to the DPL. If 
type or privilege level are found to be incompatible, the appropriate exception is generated. 


To prevent type incompatibility exceptions from being generated, software can check the access rights of a 
segment descriptor using the LAR (load access rights) instruction. The LAR instruction specifies the segment 
selector for the segment descriptor whose access rights are to be checked and a destination register. The instruc- 
tion then performs the following operations: 


1. Check that the segment selector is not null. 


5-24 Vol. 3A 


PROTECTION 


2. Checks that the segment selector points to a segment descriptor that is within the descriptor table limit (GDT 
or LDT). 


3. Checks that the segment descriptor is a code, data, LDT, call gate, task gate, or TSS segment-descriptor type. 


If the segment is not a conforming code segment, checks if the segment descriptor is visible at the CPL (that 
is, if the CPL and the RPL of the segment selector are less than or equal to the DPL). 


5. If the privilege level and type checks pass, loads the second doubleword of the segment descriptor into the 
destination register (masked by the value OOFXFFOOH, where X indicates that the corresponding 4 bits are 
undefined) and sets the ZF flag in the EFLAGS register. If the segment selector is not visible at the current 
privilege level or is an invalid type for the LAR instruction, the instruction does not modify the destination 
register and clears the ZF flag. 


Once loaded in the destination register, software can perform additional checks on the access rights information. 


5.10.2 Checking Read/Write Rights (VERR and VERW Instructions) 


When the processor accesses any code or data segment it checks the read/write privileges assigned to the 
segment to verify that the intended read or write operation is allowed. Software can check read/write rights using 
the VERR (verify for reading) and VERW (verify for writing) instructions. Both these instructions specify the 
segment selector for the segment being checked. The instructions then perform the following operations: 


1. Check that the segment selector is not null. 


2. Checks that the segment selector points to a segment descriptor that is within the descriptor table limit (GDT 
or LDT). 


Checks that the segment descriptor is a code or data-segment descriptor type. 


If the segment is not a conforming code segment, checks if the segment descriptor is visible at the CPL (that 
is, if the CPL and the RPL of the segment selector are less than or equal to the DPL). 


5. Checks that the segment is readable (for the VERR instruction) or writable (for the VERW) instruction. 


The VERR instruction sets the ZF flag in the EFLAGS register if the segment is visible at the CPL and readable; the 
VERW sets the ZF flag if the segment is visible and writable. (Code segments are never writable.) The ZF flag is 
cleared if any of these checks fail. 


5.10.3. Checking That the Pointer Offset Is Within Limits (LSL Instruction) 


When the processor accesses any segment it performs a limit check to ensure that the offset is within the limit of 
the segment. Software can perform this limit check using the LSL (load segment limit) instruction. Like the LAR 
instruction, the LSL instruction specifies the segment selector for the segment descriptor whose limit is to be 
checked and a destination register. The instruction then performs the following operations: 


1. Check that the segment selector is not null. 


2. Checks that the segment selector points to a segment descriptor that is within the descriptor table limit (GDT 
or LDT). 


3. Checks that the segment descriptor is a code, data, LDT, or TSS segment-descriptor type. 


If the segment is not a conforming code segment, checks if the segment descriptor is visible at the CPL (that 
is, if the CPL and the RPL of the segment selector less than or equal to the DPL). 


5. Ifthe privilege level and type checks pass, loads the unscrambled limit (the limit scaled according to the setting 
of the G flag in the segment descriptor) into the destination register and sets the ZF flag in the EFLAGS register. 
If the segment selector is not visible at the current privilege level or is an invalid type for the LSL instruction, 
the instruction does not modify the destination register and clears the ZF flag. 


Once loaded in the destination register, software can compare the segment limit with the offset of a pointer. 
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5.10.4 Checking Caller Access Privileges (ARPL Instruction) 


The requestor’s privilege level (RPL) field of a segment selector is intended to carry the privilege level of a calling 
procedure (the calling procedure’s CPL) to a called procedure. The called procedure then uses the RPL to determine 
if access to a segment is allowed. The RPL is said to “weaken” the privilege level of the called procedure to that of 
the RPL. 


Operating-system procedures typically use the RPL to prevent less privileged application programs from accessing 
data located in more privileged segments. When an operating-system procedure (the called procedure) receives a 
segment selector from an application program (the calling procedure), it sets the segment selector’s RPL to the 
privilege level of the calling procedure. Then, when the operating system uses the segment selector to access its 
associated segment, the processor performs privilege checks using the calling procedure’s privilege level (stored in 
the RPL) rather than the numerically lower privilege level (the CPL) of the operating-system procedure. The RPL 
thus ensures that the operating system does not access a segment on behalf of an application program unless that 
program itself has access to the segment. 


Figure 5-15 shows an example of how the processor uses the RPL field. In this example, an application program 
(located in code segment A) possesses a segment selector (segment selector D1) that points to a privileged data 
structure (that is, a data structure located in a data segment D at privilege level 0). 


The application program cannot access data segment D, because it does not have sufficient privilege, but the oper- 
ating system (located in code segment C) can. So, in an attempt to access data segment D, the application 
program executes a call to the operating system and passes segment selector D1 to the operating system as a 
parameter on the stack. Before passing the segment selector, the (well behaved) application program sets the RPL 
of the segment selector to its current privilege level (which in this example is 3). If the operating system attempts 
to access data segment D using segment selector D1, the processor compares the CPL (which is now 0 following 
the call), the RPL of segment selector D1, and the DPL of data segment D (which is 0). Since the RPL is greater than 
the DPL, access to data segment D is denied. The processor’s protection mechanism thus protects data segment D 
from access by the operating system, because application program’s privilege level (represented by the RPL of 
segment selector B) is greater than the DPL of data segment D. 


Passed as a 
parameter on 
the stack. 
Application Program \ 
Code Call 
Segment A Gate Selector B_ |»! Gate B _\ _| Segment Sel. D1 
CPL=3 RPL=3 RPL=3 
DPL=3 
Lowest Privilege 
Access 
not 
allowed 
- 
y 
; Code | =—— Data 
eens Segment C| | Segment Sel. D2 | | Segment D 
ystem = RPL=0 
DPL=0 = 
Access DEO 
0 | Highest Privilege allowed 


Figure 5-15. Use of RPL to Weaken Privilege Level of Called Procedure 
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Now assume that instead of setting the RPL of the segment selector to 3, the application program sets the RPL to 
0 (segment selector D2). The operating system can now access data segment D, because its CPL and the RPL of 
segment selector D2 are both equal to the DPL of data segment D. 


Because the application program is able to change the RPL of a segment selector to any value, it can potentially use 
a procedure operating at a numerically lower privilege level to access a protected data structure. This ability to 
lower the RPL of a segment selector breaches the processor’s protection mechanism. 


Because a called procedure cannot rely on the calling procedure to set the RPL correctly, operating-system proce- 
dures (executing at numerically lower privilege-levels) that receive segment selectors from numerically higher 
privilege-level procedures need to test the RPL of the segment selector to determine if it is at the appropriate level. 
The ARPL (adjust requested privilege level) instruction is provided for this purpose. This instruction adjusts the RPL 
of one segment selector to match that of another segment selector. 


The example in Figure 5-15 demonstrates how the ARPL instruction is intended to be used. When the operating- 
system receives segment selector D2 from the application program, it uses the ARPL instruction to compare the 
RPL of the segment selector with the privilege level of the application program (represented by the code-segment 
selector pushed onto the stack). If the RPL is less than application program’s privilege level, the ARPL instruction 
changes the RPL of the segment selector to match the privilege level of the application program (segment selector 
D1). Using this instruction thus prevents a procedure running at a numerically higher privilege level from 
accessing numerically lower privilege-level (more privileged) segments by lowering the RPL of a segment selector. 


Note that the privilege level of the application program can be determined by reading the RPL field of the segment 
selector for the application-program’s code segment. This segment selector is stored on the stack as part of the call 
to the operating system. The operating system can copy the segment selector from the stack into a register for 
use as an operand for the ARPL instruction. 


5.10.5 Checking Alignment 


When the CPL is 3, alignment of memory references can be checked by setting the AM flag in the CRO register and 
the AC flag in the EFLAGS register. Unaligned memory references generate alignment exceptions (#AC). The 
processor does not generate alignment exceptions when operating at privilege level 0, 1, or 2. See Table 6-7 fora 
description of the alignment requirements when alignment checking is enabled. 


5.11 PAGE-LEVEL PROTECTION 


Page-level protection can be used alone or applied to segments. When page-level protection is used with the flat 
memory model, it allows supervisor code and data (the operating system or executive) to be protected from user 
code and data (application programs). It also allows pages containing code to be write protected. When the 
segment- and page-level protection are combined, page-level read/write protection allows more protection granu- 
larity within segments. 


With page-level protection (as with segment-level protection) each memory reference is checked to verify that 
protection checks are satisfied. All checks are made before the memory cycle is started, and any violation prevents 
the cycle from starting and results in a page-fault exception being generated. Because checks are performed in 
parallel with address translation, there is no performance penalty. 


The processor performs two page-level protection checks: 
¢ Restriction of addressable domain (supervisor and user modes). 
° Page type (read only or read/write). 


Violations of either of these checks results in a page-fault exception being generated. See Chapter 6, “Interrupt 
14—Page-Fault Exception (#PF),” for an explanation of the page-fault exception mechanism. This chapter 
describes the protection violations which lead to page-fault exceptions. 
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5.11.1 Page-Protection Flags 


Protection information for pages is contained in two flags in a paging-structure entry (see Chapter 4): the 
read/write flag (bit 1) and the user/supervisor flag (bit 2). The protection checks use the flags in all paging struc- 
tures. 


5.11.2 Restricting Addressable Domain 


The page-level protection mechanism allows restricting access to pages based on two privilege levels: 


® Supervisor mode (U/S flag is 0)—(Most privileged) For the operating system or executive, other system 
software (such as device drivers), and protected system data (such as page tables). 


e User mode (U/S flag is 1)—(Least privileged) For application code and data. 


The segment privilege levels map to the page privilege levels as follows. If the processor is currently operating at 
a CPL of 0, 1, or 2, it is in Supervisor mode; if it is operating at a CPL of 3, it is in user mode. When the processor is 
in supervisor mode, it can access all pages; when in user mode, it can access only user-level pages. (Note that the 
WP flag in control register CRO modifies the supervisor permissions, as described in Section 5.11.3, “Page Type.”) 


Note that to use the page-level protection mechanism, code and data segments must be set up for at least two 
segment-based privilege levels: level 0 for supervisor code and data segments and level 3 for user code and data 
segments. (In this model, the stacks are placed in the data segments.) To minimize the use of segments, a flat 
memory model can be used (see Section 3.2.1, “Basic Flat Model”). 


Here, the user and supervisor code and data segments all begin at address zero in the linear address space and 
overlay each other. With this arrangement, operating-system code (running at the supervisor level) and application 
code (running at the user level) can execute as if there are no segments. Protection between operating-system and 
application code and data is provided by the processor’s page-level protection mechanism. 


5.11.3. Page Type 


The page-level protection mechanism recognizes two page types: 
® Read-only access (R/W flag is 0). 
° Read/write access (R/W flag is 1). 


When the processor is in supervisor mode and the WP flag in register CRO is clear (its state following reset initial- 
ization), all pages are both readable and writable (write-protection is ignored). When the processor is in user mode, 
it can write only to user-mode pages that are read/write accessible. User-mode pages which are read/write or 
read-only are readable; supervisor-mode pages are neither readable nor writable from user mode. A page-fault 
exception is generated on any attempt to violate the protection rules. 


Starting with the P6 family, Intel processors allow user-mode pages to be write-protected against supervisor-mode 
access. Setting CRO.WP = 1 enables supervisor-mode sensitivity to write protected pages. If CRO.WP = 1, read- 
only pages are not writable from any privilege level. This supervisor write-protect feature is useful for imple- 
menting a “copy-on-write” strategy used by some operating systems, such as UNIX*, for task creation (also called 
forking or spawning). When a new task is created, it is possible to copy the entire address space of the parent task. 
This gives the child task a complete, duplicate set of the parent's segments and pages. An alternative copy-on- 
write strategy saves memory space and time by mapping the child's segments and pages to the same segments 
and pages used by the parent task. A private copy of a page gets created only when one of the tasks writes to the 
page. By using the WP flag and marking the shared pages as read-only, the supervisor can detect an attempt to 
write to a page, and can copy the page at that time. 


5.11.4 Combining Protection of Both Levels of Page Tables 


For any one page, the protection attributes of its page-directory entry (first-level page table) may differ from those 
of its page-table entry (second-level page table). The processor checks the protection for a page in both its page- 
directory and the page-table entries. Table 5-3 shows the protection provided by the possible combinations of 
protection attributes when the WP flag is clear. 
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5.11.5 Overrides to Page Protection 


The following types of memory accesses are checked as if they are privilege-level 0 accesses, regardless of the CPL 
at which the processor is currently operating: 


¢ Access to segment descriptors in the GDT, LDT, or IDT. 


¢ Access to an inner-privilege-level stack during an inter-privilege-level call or a call to in exception or interrupt 
handler, when a change of privilege level occurs. 


5.12 COMBINING PAGE AND SEGMENT PROTECTION 


When paging is enabled, the processor evaluates segment protection first, then evaluates page protection. If the 
processor detects a protection violation at either the segment level or the page level, the memory access is not 

carried out and an exception is generated. If an exception is generated by segmentation, no paging exception is 
generated. 


Page-level protections cannot be used to override segment-level protection. For example, a code segment is by 
definition not writable. If a code segment is paged, setting the R/W flag for the pages to read-write does not make 
the pages writable. Attempts to write into the pages will be blocked by segment-level protection checks. 


Page-level protection can be used to enhance segment-level protection. For example, if a large read-write data 
segment is paged, the page-protection mechanism can be used to write-protect individual pages. 


Table 5-3. Combined Page-Directory and Page-Table Protection 


Page-Directory Entry Page-Table Entry Combined Effect 

Privilege Access Type Privilege Access Type Privilege Access Type 
User Read-Only User Read-Only User Read-Only 
User Read-Only User Read-Write User Read-Only 
User Read-Write User Read-Only User Read-Only 
User Read-Write User Read-Write User Read/Write 
User Read-Only Supervisor Read-Only Supervisor Read/Write* 
User Read-Only Supervisor Read-Write Supervisor Read/Write* 
User Read-Write Supervisor Read-Only Supervisor Read/Write* 
User Read-Write Supervisor Read-Write Supervisor Read/Write 
Supervisor Read-Only User Read-Only Supervisor Read/Write* 
Supervisor Read-Only User Read-Write Supervisor Read/Write* 
Supervisor Read-Write User Read-Only Supervisor Read/Write* 
Supervisor Read-Write User Read-Write Supervisor Read/Write 
Supervisor Read-Only Supervisor Read-Only Supervisor Read/Write* 
Supervisor Read-Only Supervisor Read-Write Supervisor Read/Write* 
Supervisor Read-Write Supervisor Read-Only Supervisor Read/Write* 
Supervisor Read-Write Supervisor Read-Write Supervisor Read/Write 
NOTE: 


* If CRO.WP = 1, access type is determined by the R/W flags of the page-directory and page-table entries. IF CRO.WP = 0, supervisor 
privilege permits read-write access. 
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5.13 PAGE-LEVEL PROTECTION AND EXECUTE-DISABLE BIT 


In addition to page-level protection offered by the U/S and R/W flags, paging structures used with PAE paging, 4- 
level paging,! and 5-level paging provide the execute-disable bit (see Chapter 4, “Paging”). This bit offers addi- 
tional protection for data pages. 


An Intel 64 or IA-32 processor with the execute-disable bit capability can prevent data pages from being used by 
malicious software to execute code. This capability is provided in: 


¢ 32-bit protected mode with PAE enabled. 
e = JTA-32e mode. 


While the execute-disable bit capability does not introduce new instructions, it does require operating systems to 
use a PAE-enabled environment and establish a page-granular protection policy for memory pages. 


If the execute-disable bit of a memory page is set, that page can be used only as data. An attempt to execute code 
from a memory page with the execute-disable bit set causes a page-fault exception. 


The execute-disable capability is not supported with 32-bit paging. Existing page-level protection mechanisms (see 
Section 5.11, “Page-Level Protection”) continue to apply to memory pages independent of the execute-disable 
setting. 


5.13.1. Detecting and Enabling the Execute-Disable Capability 


Software can detect the presence of the execute-disable capability using the CPUID instruction. 
CPUID.80000001H:EDX.NX [bit 20] = 1 indicates the capability is available. 


If the capability is available, software can enable it by setting IA32_EFER.NXE[bit 11] to 1. IA32_EFER is available 
if CPUID.80000001H.EDX[bit 20 or 29] = 1. 


If the execute-disable capability is not available, a write to set IA32_EFER.NXE produces a #GP exception. See 
Table 5-4. 


Table 5-4. Extended Feature Enable MSR (IA32_EFER) 


63:12 11 10 9 8 7:1 0 
Reserved Execute-disable bit | |A-32e mode Reserved IA-32e mode Reserved SysCall enable (SCE) 
enable (NXE) active (LMA) enable (LME) 


5.13.2 Execute-Disable Page Protection 


The execute-disable bit in the paging structures enhances page protection for data pages. Instructions cannot be 
fetched from a memory page if IA32_EFER.NXE =1 and the execute-disable bit is set in any of the paging-structure 
entries used to map the page. Table 5-5 lists the valid usage of a page in relation to the value of execute-disable bit 
(bit 63) of the corresponding entry in each level of the paging structures. Execute-disable protection can be acti- 

vated using the execute-disable bit at any level of the paging structure, irrespective of the corresponding entry in 
other levels. When execute-disable protection is not activated, the page can be used as code or data. 


1. Earlier versions of this manual used the term “IA-32e paging” to identify 4-level paging. 
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Table 5-5. Page Level Protection Matrix with Execute-Disable Bit Capability with 4-Level Paging 


Execute Disable Bit Value (Bit 63) Valid Usage 
PML4 PDP PDE PTE 

Bit 63 = 1 . % i Data 

is Bit 63 = 1 * i Data 

* 5 Bit 63 = 1 : Data 

= id . Bit 63 = 1 Data 

Bit 63 = 0 Bit 63 =O Bit 63 = 0 Bit 63 = 0 Data/Code 
NOTES: 


* Value not checked. 


In legacy PAE-enabled mode, Table 5-6 and Table 5-7 show the effect of setting the execute-disable bit for code 
and data pages. 


Table 5-6. 4-KByte Page Level Protection Matrix with Execute-Disable Bit Capability with PAE Paging 


Execute Disable Bit Value (Bit 63) Valid Usage 
PDE PTE 

Bit 63 = 1 * Data 

: Bit 63 = 1 Data 

Bit 63 = 0 Bit 63 = 0 Data/Code 
NOTE: 


* Value not checked. 


Table 5-7. 2-MByte Page Level Protection with Execute-Disable Bit Capability with PAE Paging 


Execute Disable Bit Value (Bit 63) Valid Usage 
PDE 

Bit 63 = 1 Data 

Bit 63 =O Data/Code 


5.13.3. Reserved Bit Checking 


The processor enforces reserved bit checking in paging data structure entries. The bits being checked varies with 
paging mode and may vary with the size of physical address space. 


Table 5-8 shows the reserved bits that are checked when the execute disable bit capability is enabled (CR4.PAE = 1 
and IA32_EFER.NXE = 1). Table 5-8 and Table 5-9 show the following paging modes: 


* Non-PAE 4-KByte paging: 4-KByte-page only paging (CR4.PAE = 0, CR4.PSE = 0). 
° PSE36: 4-KByte and 4-MByte pages (CR4.PAE = 0, CR4.PSE = 1). 
¢ PAE: 4-KByte and 2-MByte pages (CR4.PAE = 1, CR4.PSE = X). 


The reserved bit checking depends on the physical address size supported by the implementation, which is 
reported in CPUID.80000008H. See the table note. 
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Table 5-8. Page Level Protection Matrix with Execute-Disable Bit Capability Enabled 


Mode Paging Mode Check Bits 
32-bit 4-KByte paging (non-PAE) No reserved bits checked 
PSE36 - PDE, 4-MByte page Bit [21] 
PSE36 - PDE, 4-KByte page No reserved bits checked 
PSE36 - PTE No reserved bits checked 
PAE - PDP table entry Bits [63:MAXPHYADDR] & [8:5] & [2:1] * 
PAE - PDE, 2-MByte page Bits [62:MAXPHYADDR] & [20:13] * 
PAE - PDE, 4-KByte page Bits [62:MAXPHYADDR] * 
PAE - PTE Bits [62:MAXPHYADDR] * 
64-bit PML5E Bits [5 1:MAXPHYADDR] * 
PML4E Bits [5 1:MAXPHYADDR] * 
PDPTE Bits [5 1:MAXPHYADDR] * 
PDE, 2-MByte page Bits [51:MAXPHYADDR] & [20:13] * 
PDE, 4-KByte page Bits [51:MAXPHYADDR] * 
PTE Bits [51:MAXPHYADDR] * 
NOTES: 


* MAXPHYADDR is the maximum physical address size and is indicated by CPUID.BO000008H:EAX [bits 7-0]. 


If execute disable bit capability is not enabled or not available, reserved bit checking in 64-bit mode includes bit 63 
and additional bits. This and reserved bit checking for legacy 32-bit paging modes are shown in Table 5-10. 


Table 5-9. Reserved Bit Checking With Execute-Disable Bit Capability Not Enabled 


Mode Paging Mode Check Bits 
32-bit KByte paging (non-PAE) No reserved bits checked 
PSE36 - PDE, 4-MByte page Bit [21] 
PSE36 - PDE, 4-KByte page No reserved bits checked 
PSE36 - PTE No reserved bits checked 
PAE - PDP table entry Bits [63:MAXPHYADDR] & [8:5] & [2:1]* 
PAE - PDE, 2-MByte page Bits [63:MAXPHYADDR] & [20:13]* 
PAE - PDE, 4-KByte page Bits [63:-MAXPHYADDR]* 
PAE - PTE Bits [63:MAXPHYADDR]* 
64-bit PML5E Bit [63], bits [51:MAXPHYADDR]* 
PML4E Bit [63], bits [51:MAXPHYADDR]* 
PDPTE Bit [63], bits [51:MAXPHYADDR]* 
PDE, 2-MByte page Bit [63], bits [51:MAXPHYADDR] & [20:1 3]* 
PDE, 4-KByte page Bit [63], bits [51:MAXPHYADDR]* 
PTE Bit [63], bits [51:MAXPHYADDR]* 
NOTES: 


* MAXPHYADDR is the maximum physical address size and is indicated by CPUID.BO000008H:EAX [bits 7-0]. 


5.13.4 Exception Handling 


When execute disable bit capability is enabled (IA32_EFER.NXE = 1), conditions for a page fault to occur include 
the same conditions that apply to an Intel 64 or IA-32 processor without execute disable bit capability plus the 
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following new condition: an instruction fetch to a linear address that translates to physical address in a memory 
page that has the execute-disable bit set. 


An Execute Disable Bit page fault can occur at all privilege levels. It can occur on any instruction fetch, including 
(but not limited to): near branches, far branches, CALL/RET/INT/IRET execution, sequential instruction fetches, 
and task switches. The execute-disable bit in the page translation mechanism is checked only when: 


¢ = JTA32_EFER.NXE = 1. 
¢ The instruction translation look-aside buffer (ITLB) is loaded with a page that is not already present in the ITLB. 
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CHAPTER 6 
INTERRUPT AND EXCEPTION HANDLING 


This chapter describes the interrupt and exception-handling mechanism when operating in protected mode on an 
Intel 64 or IA-32 processor. Most of the information provided here also applies to interrupt and exception mecha- 
nisms used in real-address, virtual-8086 mode, and 64-bit mode. 


Chapter 19, “8086 Emulation,” describes information specific to interrupt and exception mechanisms in real- 
address and virtual-8086 mode. Section 6.14, “Exception and Interrupt Handling in 64-bit Mode,” describes infor- 
mation specific to interrupt and exception mechanisms in IA-32e mode and 64-bit sub-mode. 


6.1 INTERRUPT AND EXCEPTION OVERVIEW 


Interrupts and exceptions are events that indicate that a condition exists somewhere in the system, the processor, 
or within the currently executing program or task that requires the attention of a processor. They typically result in 
a forced transfer of execution from the currently running program or task to a special software routine or task 
called an interrupt handler or an exception handler. The action taken by a processor in response to an interrupt or 
exception is referred to as servicing or handling the interrupt or exception. 


Interrupts occur at random times during the execution of a program, in response to signals from hardware. System 
hardware uses interrupts to handle events external to the processor, such as requests to service peripheral devices. 
Software can also generate interrupts by executing the INT rn instruction. 


Exceptions occur when the processor detects an error condition while executing an instruction, such as division by 
zero. The processor detects a variety of error conditions including protection violations, page faults, and internal 
machine faults. The machine-check architecture of the Pentium 4, Intel Xeon, P6 family, and Pentium processors 
also permits a machine-check exception to be generated when internal hardware errors and bus errors are 
detected. 


When an interrupt is received or an exception is detected, the currently running procedure or task is suspended 
while the processor executes an interrupt or exception handler. When execution of the handler is complete, the 
processor resumes execution of the interrupted procedure or task. The resumption of the interrupted procedure or 
task happens without loss of program continuity, unless recovery from an exception was not possible or an inter- 
rupt caused the currently running program to be terminated. 


This chapter describes the processor’s interrupt and exception-handling mechanism, when operating in protected 
mode. A description of the exceptions and the conditions that cause them to be generated is given at the end of this 
chapter. 


6.2 EXCEPTION AND INTERRUPT VECTORS 


To aid in handling exceptions and interrupts, each architecturally defined exception and each interrupt condition 
requiring special handling by the processor is assigned a unique identification number, called a vector number. The 
processor uses the vector number assigned to an exception or interrupt as an index into the interrupt descriptor 
table (IDT). The table provides the entry point to an exception or interrupt handler (see Section 6.10, “Interrupt 
Descriptor Table (IDT)”). 


The allowable range for vector numbers is 0 to 255. Vector numbers in the range 0 through 31 are reserved by the 
Intel 64 and IA-32 architectures for architecture-defined exceptions and interrupts. Not all of the vector numbers 
in this range have a currently defined function. The unassigned vector numbers in this range are reserved. Do not 
use the reserved vector numbers. 


Vector numbers in the range 32 to 255 are designated as user-defined interrupts and are not reserved by the Intel 
64 and IA-32 architecture. These interrupts are generally assigned to external I/O devices to enable those devices 
to send interrupts to the processor through one of the external hardware interrupt mechanisms (see Section 6.3, 
“Sources of Interrupts”). 
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Table 6-1 shows vector number assignments for architecturally defined exceptions and for the NMI interrupt. This 
table gives the exception type (see Section 6.5, “Exception Classifications”) and indicates whether an error code is 
saved on the stack for the exception. The source of each predefined exception and the NMI interrupt is also given. 


6.3 SOURCES OF INTERRUPTS 


The processor receives interrupts from two sources: 
¢ External (hardware generated) interrupts. 
° Software-generated interrupts. 


6.3.1 External Interrupts 


External interrupts are received through pins on the processor or through the local APIC. The primary interrupt pins 
on Pentium 4, Intel Xeon, P6 family, and Pentium processors are the LINT[1:0] pins, which are connected to the 
local APIC (see Chapter 10, “Advanced Programmable Interrupt Controller (APIC)”). When the local APIC is 
enabled, the LINT[1:0] pins can be programmed through the APIC’s local vector table (LVT) to be associated with 
any of the processor's exception or interrupt vectors. 


When the local APIC is global/hardware disabled, these pins are configured as INTR and NMI pins, respectively. 
Asserting the INTR pin signals the processor that an external interrupt has occurred. The processor reads from the 
system bus the interrupt vector number provided by an external interrupt controller, such as an 8259A (see Section 
6.2, “Exception and Interrupt Vectors”). Asserting the NMI pin signals a non-maskable interrupt (NMI), which is 
assigned to interrupt vector 2. 


Table 6-1. Protected-Mode Exceptions and Interrupts 


Vector | Mnemonic | Description Type Error Source 
Code 
0 #DE Divide Error Fault No DIV and IDIV instructions. 
#DB Debug Exception Fault/ Trap No Instruction, data, and I/O breakpoints; 

single-step; and others. 

2 - NMI Interrupt Interrupt No Nonmaskable external interrupt. 

3 #BP Breakpoint Trap No INT3 instruction. 

4 HOF Overflow Trap No INTO instruction. 

5 #BR BOUND Range Exceeded Fault No BOUND instruction. 

6 #UD Invalid Opcode (Undefined Opcode) Fault No UD instruction or reserved opcode. 

7 #NM Device Not Available (No Math Fault No Floating-point or WAIT/FWAIT instruction. 

Coprocessor) 
8 #DF Double Fault Abort Yes Any instruction that can generate an 
(zero) exception, an NMI, or an INTR. 
9 Coprocessor Segment Overrun Fault No Floating-point instruction. ! 
(reserved) 

10 #TS Invalid TSS Fault Yes Task switch or TSS access. 

11 #NP Segment Not Present Fault Yes Loading segment registers or accessing 
system segments. 

12 #SS Stack-Segment Fault Fault Yes Stack operations and SS register loads. 

13 #GP General Protection Fault Yes Any memory reference and other 
protection checks. 

14 HPF Page Fault Fault Yes Any memory reference. 
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Table 6-1. Protected-Mode Exceptions and Interrupts (Contd.) 


Vector | Mnemonic | Description Type Error Source 
Code 
15 _ (Intel reserved. Do not use.) No 
16 #MF x87 FPU Floating-Point Error (Math | Fault No x87 FPU floating-point or WAIT/FWAIT 
Fault) instruction. 
17 #AC Alignment Check Fault Yes Any data reference in memory.¢ 
(Zero) 

18 #MC Machine Check Abort No Error codes (if any) and source are model 
dependent. 

19 #XM SIMD Floating-Point Exception Fault No SSE/SSE2/SSE3 floating-point 
instructions* 

20 #VE Virtualization Exception Fault No EPT violations? 

21 #CP Control Protection Exception Fault Yes RET, IRET, RSTORSSP, and SETSSBSY 
instructions can generate this exception. 
When CET indirect branch tracking is 
enabled, this exception can be generated 
due to a missing ENDBRANCH instruction 
at target of an indirect call or jump. 

22-31 |— Intel reserved. Do not use. 

32-255 | — User Defined (Non-reserved) Interrupt External interrupt or INT n instruction. 

Interrupts 
NOTES: 


1. Processors after the Intel386 processor do not generate this exception. 

2. This exception was introduced in the Intel486 processor. 

3. This exception was introduced in the Pentium processor and enhanced in the P6 family processors. 

4. This exception was introduced in the Pentium Ill processor. 

5. This exception can occur only on processors that support the 1-setting of the “EPT-violation #VE" VM-execution control. 


The processor's local APIC is normally connected to a system-based I/O APIC. Here, external interrupts received at 
the I/O APIC’s pins can be directed to the local APIC through the system bus (Pentium 4, Intel Core Duo, Intel Core 
2, Intel Atom®, and Intel Xeon processors) or the APIC serial bus (P6 family and Pentium processors). The I/O APIC 
determines the vector number of the interrupt and sends this number to the local APIC. When a system contains 
multiple processors, processors can also send interrupts to one another by means of the system bus (Pentium 4, 
Intel Core Duo, Intel Core 2, Intel Atom, and Intel Xeon processors) or the APIC serial bus (P6 family and Pentium 
processors). 


The LINT[1:0] pins are not available on the Intel486 processor and earlier Pentium processors that do not contain 
an on-chip local APIC. These processors have dedicated NMI and INTR pins. With these processors, external inter- 
rupts are typically generated by a system-based interrupt controller (8259A), with the interrupts being signaled 
through the INTR pin. 


Note that several other pins on the processor can cause a processor interrupt to occur. However, these interrupts 
are not handled by the interrupt and exception mechanism described in this chapter. These pins include the 
RESET#, FLUSH#, STPCLK#, SMI#, R/S#, and INIT# pins. Whether they are included on a particular processor is 
implementation dependent. Pin functions are described in the data books for the individual processors. The SMI# 
pin is described in Chapter 30, “System Management Mode.” 


6.3.2 Maskable Hardware Interrupts 


Any external interrupt that is delivered to the processor by means of the INTR pin or through the local APIC is called 
a maskable hardware interrupt. Maskable hardware interrupts that can be delivered through the INTR pin include 
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all IA-32 architecture defined interrupt vectors from O through 255; those that can be delivered through the local 
APIC include interrupt vectors 16 through 255. 


The IF flag in the EFLAGS register permits all maskable hardware interrupts to be masked as a group (see Section 
6.8.1, “Masking Maskable Hardware Interrupts”). Note that when interrupts 0 through 15 are delivered through the 
local APIC, the APIC indicates the receipt of an illegal vector. 


6.3.3 Software-Generated Interrupts 


The INT rn instruction permits interrupts to be generated from within software by supplying an interrupt vector 
number as an operand. For example, the INT 35 instruction forces an implicit call to the interrupt handler for inter- 
rupt 35. 


Any of the interrupt vectors from 0 to 255 can be used as a parameter in this instruction. If the processor’s 
predefined NMI vector is used, however, the response of the processor will not be the same as it would be from an 
NMI interrupt generated in the normal manner. If vector number 2 (the NMI vector) is used in this instruction, the 
NMI interrupt handler is called, but the processor’s NMI-handling hardware is not activated. 


Interrupts generated in software with the INT rn instruction cannot be masked by the IF flag in the EFLAGS register. 


6.4 SOURCES OF EXCEPTIONS 


The processor receives exceptions from three sources: 
¢ Processor-detected program-error exceptions. 

° Software-generated exceptions. 

¢ Machine-check exceptions. 


6.4.1 Program-Error Exceptions 


The processor generates one or more exceptions when it detects program errors during the execution in an appli- 
cation program or the operating system or executive. Intel 64 and IA-32 architectures define a vector number for 
each processor-detectable exception. Exceptions are classified as faults, traps, and aborts (see Section 6.5, 
“Exception Classifications”). 


6.4.2 Software-Generated Exceptions 


The INTO, INT1, INT3, and BOUND instructions permit exceptions to be generated in software. These instructions 
allow checks for exception conditions to be performed at points in the instruction stream. For example, INT3 causes 
a breakpoint exception to be generated. 


The INT nr instruction can be used to emulate exceptions in software; but there is a limitation.! If INT n provides a 
vector for one of the architecturally-defined exceptions, the processor generates an interrupt to the correct vector 
(to access the exception handler) but does not push an error code on the stack. This is true even if the associated 
hardware-generated exception normally produces an error code. The exception handler will still attempt to pop an 
error code from the stack while handling the exception. Because no error code was pushed, the handler will pop off 
and discard the EIP instead (in place of the missing error code). This sends the return to the wrong location. 


6.4.3 Machine-Check Exceptions 


The P6 family and Pentium processors provide both internal and external machine-check mechanisms for checking 
the operation of the internal chip hardware and bus transactions. These mechanisms are implementation depen- 


1. The INT ninstruction has opcode CD following by an immediate byte encoding the value of n. In contrast, INT1 has opcode F1 and 
INT3 has opcode CC. 
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dent. When a machine-check error is detected, the processor signals a machine-check exception (vector 18) and 
returns an error code. 


See Chapter 6, “Interrupt 18—Machine-Check Exception (#MC)” and Chapter 15, “Machine-Check Architecture,” 
for more information about the machine-check mechanism. 


6.5 EXCEPTION CLASSIFICATIONS 


Exceptions are classified as faults, traps, or aborts depending on the way they are reported and whether the 
instruction that caused the exception can be restarted without loss of program or task continuity. 


¢ Faults — A fault is an exception that can generally be corrected and that, once corrected, allows the program 
to be restarted with no loss of continuity. When a fault is reported, the processor restores the machine state to 
the state prior to the beginning of execution of the faulting instruction. The return address (saved contents of 
the CS and EIP registers) for the fault handler points to the faulting instruction, rather than to the instruction 
following the faulting instruction. 


¢ Traps — Atrap is an exception that is reported immediately following the execution of the trapping instruction. 
Traps allow execution of a program or task to be continued without loss of program continuity. The return 
address for the trap handler points to the instruction to be executed after the trapping instruction. 


¢ Aborts — An abort is an exception that does not always report the precise location of the instruction causing 
the exception and does not allow a restart of the program or task that caused the exception. Aborts are used to 
report severe errors, such as hardware errors and inconsistent or illegal values in system tables. 


NOTE 


One exception subset normally reported as a fault is not restartable. Such exceptions result in loss 
of some processor state. For example, executing a POPAD instruction where the stack frame 
crosses over the end of the stack segment causes a fault to be reported. In this situation, the 
exception handler sees that the instruction pointer (CS:EIP) has been restored as if the POPAD 
instruction had not been executed. However, internal processor state (the general-purpose 
registers) will have been modified. Such cases are considered programming errors. An application 
causing this class of exceptions should be terminated by the operating system. 


6.6 PROGRAM OR TASK RESTART 


To allow the restarting of program or task following the handling of an exception or an interrupt, all exceptions 
(except aborts) are guaranteed to report exceptions on an instruction boundary. All interrupts are guaranteed to be 
taken on an instruction boundary. 


For fault-class exceptions, the return instruction pointer (saved when the processor generates an exception) points 
to the faulting instruction. So, when a program or task is restarted following the handling of a fault, the faulting 
instruction is restarted (re-executed). Restarting the faulting instruction is commonly used to handle exceptions 
that are generated when access to an operand is blocked. The most common example of this type of fault is a page- 
fault exception (#PF) that occurs when a program or task references an operand located on a page that is not in 
memory. When a page-fault exception occurs, the exception handler can load the page into memory and resume 
execution of the program or task by restarting the faulting instruction. To ensure that the restart is handled trans- 
parently to the currently executing program or task, the processor saves the necessary registers and stack pointers 
to allow a restart to the state prior to the execution of the faulting instruction. 


For trap-class exceptions, the return instruction pointer points to the instruction following the trapping instruction. 
If a trap is detected during an instruction which transfers execution, the return instruction pointer reflects the 
transfer. For example, if a trap is detected while executing a JMP instruction, the return instruction pointer points 
to the destination of the JMP instruction, not to the next address past the JMP instruction. All trap exceptions allow 
program or task restart with no loss of continuity. For example, the overflow exception is a trap exception. Here, 
the return instruction pointer points to the instruction following the INTO instruction that tested EFLAGS.OF (over- 
flow) flag. The trap handler for this exception resolves the overflow condition. Upon return from the trap handler, 
program or task execution continues at the instruction following the INTO instruction. 
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The abort-class exceptions do not support reliable restarting of the program or task. Abort handlers are designed 
to collect diagnostic information about the state of the processor when the abort exception occurred and then shut 
down the application and system as gracefully as possible. 


Interrupts rigorously support restarting of interrupted programs and tasks without loss of continuity. The return 
instruction pointer saved for an interrupt points to the next instruction to be executed at the instruction boundary 
where the processor took the interrupt. If the instruction just executed has a repeat prefix, the interrupt is taken 
at the end of the current iteration with the registers set to execute the next iteration. 


The ability of a P6 family processor to speculatively execute instructions does not affect the taking of interrupts by 
the processor. Interrupts are taken at instruction boundaries located during the retirement phase of instruction 
execution; so they are always taken in the “in-order” instruction stream. See Chapter 2, “Intel® 64 and IA-32 
Architectures,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, for more infor- 
mation about the P6 family processors’ microarchitecture and its support for out-of-order instruction execution. 


Note that the Pentium processor and earlier IA-32 processors also perform varying amounts of prefetching and 
preliminary decoding. With these processors as well, exceptions and interrupts are not signaled until actual “in- 
order” execution of the instructions. For a given code sample, the signaling of exceptions occurs uniformly when 
the code is executed on any family of IA-32 processors (except where new exceptions or new opcodes have been 
defined). 


6.7 NONMASKABLE INTERRUPT (NMI) 


The nonmaskable interrupt (NMI) can be generated in either of two ways: 
¢ External hardware asserts the NMI pin. 


¢ The processor receives a message on the system bus (Pentium 4, Intel Core Duo, Intel Core 2, Intel Atom, and 
Intel Xeon processors) or the APIC serial bus (P6 family and Pentium processors) with a delivery mode NMI. 


When the processor receives a NMI from either of these sources, the processor handles it immediately by calling 
the NMI handler pointed to by interrupt vector number 2. The processor also invokes certain hardware conditions 
to ensure that no other interrupts, including NMI interrupts, are received until the NMI handler has completed 
executing (see Section 6.7.1, “Handling Multiple NMIs”). 


Also, when an NMI is received from either of the above sources, it cannot be masked by the IF flag in the EFLAGS 
register. 


It is possible to issue a maskable hardware interrupt (through the INTR pin) to vector 2 to invoke the NMI interrupt 
handler; however, this interrupt will not truly be an NMI interrupt. A true NMI interrupt that activates the 
processor’s NMI-handling hardware can only be delivered through one of the mechanisms listed above. 


6.7.1 Handling Multiple NMIs 


While an NMI interrupt handler is executing, the processor blocks delivery of subsequent NMIs until the next execu- 
tion of the IRET instruction. This blocking of NMIs prevents nested execution of the NMI handler. It is recommended 
that the NMI interrupt handler be accessed through an interrupt gate to disable maskable hardware interrupts (see 
Section 6.8.1, “Masking Maskable Hardware Interrupts”). 


An execution of the IRET instruction unblocks NMIs even if the instruction causes a fault. For example, if the IRET 
instruction executes with EFLAGS.VM = 1 and IOPL of less than 3, a general-protection exception is generated (see 
Section 19.2.7, “Sensitive Instructions”). In such a case, NMIs are unmasked before the exception handler is 
invoked. 


6.8 ENABLING AND DISABLING INTERRUPTS 


The processor inhibits the generation of some interrupts, depending on the state of the processor and of the IF and 
RF flags in the EFLAGS register, as described in the following sections. 
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6.8.1 Masking Maskable Hardware Interrupts 


The IF flag can disable the servicing of maskable hardware interrupts received on the processor’s INTR pin or 
through the local APIC (see Section 6.3.2, “Maskable Hardware Interrupts”). When the IF flag is clear, the 
processor inhibits interrupts delivered to the INTR pin or through the local APIC from generating an internal inter- 
rupt request; when the IF flag is set, interrupts delivered to the INTR or through the local APIC pin are processed 
as normal external interrupts. 


The IF flag does not affect non-maskable interrupts (NMIs) delivered to the NMI pin or delivery mode NMI 
messages delivered through the local APIC, nor does it affect processor generated exceptions. As with the other 
flags in the EFLAGS register, the processor clears the IF flag in response to a hardware reset. 


The fact that the group of maskable hardware interrupts includes the reserved interrupt and exception vectors 0 
through 32 can potentially cause confusion. Architecturally, when the IF flag is set, an interrupt for any of the 
vectors from 0 through 32 can be delivered to the processor through the INTR pin and any of the vectors from 16 
through 32 can be delivered through the local APIC. The processor will then generate an interrupt and call the 
interrupt or exception handler pointed to by the vector number. So for example, it is possible to invoke the page- 
fault handler through the INTR pin (by means of vector 14); however, this is not a true page-fault exception. It is 
an interrupt. As with the INT rn instruction (see Section 6.4.2, “Software-Generated Exceptions”), when an inter- 
rupt is generated through the INTR pin to an exception vector, the processor does not push an error code on the 
stack, so the exception handler may not operate correctly. 


The IF flag can be set or cleared with the STI (set interrupt-enable flag) and CLI (clear interrupt-enable flag) 
instructions, respectively. These instructions may be executed only if the CPL is equal to or less than the IOPL. A 
general-protection exception (#GP) is generated if they are executed when the CPL is greater than the IOPL.? If 
IF = 0, maskable hardware interrupts remain inhibited on the instruction boundary following an execution of STI.? 
The inhibition ends after delivery of another event (e.g., exception) or the execution of the next instruction. 


The IF flag is also affected by the following operations: 


¢ The PUSHF instruction stores all flags on the stack, where they can be examined and modified. The POPF 
instruction can be used to load the modified flags back into the EFLAGS register. 


¢ Task switches and the POPF and IRET instructions load the EFLAGS register; therefore, they can be used to 
modify the setting of the IF flag. 


¢ =When an interrupt is handled through an interrupt gate, the IF flag is automatically cleared, which disables 
maskable hardware interrupts. (If an interrupt is handled through a trap gate, the IF flag is not cleared.) 


See the descriptions of the CLI, STI, PUSHF, POPF, and IRET instructions in Chapter 3, “Instruction Set Reference, 
A-L,” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A, and Chapter 4, “Instruc- 
tion Set Reference, M-U,” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2B, for a 
detailed description of the operations these instructions are allowed to perform on the IF flag. 


6.8.2 Masking Instruction Breakpoints 


The RF (resume) flag in the EFLAGS register controls the response of the processor to instruction-breakpoint condi- 
tions (see the description of the RF flag in Section 2.3, “System Flags and Fields in the EFLAGS Register”). 


When set, it prevents an instruction breakpoint from generating a debug exception (#DB); when clear, instruction 
breakpoints will generate debug exceptions. The primary function of the RF flag is to prevent the processor from 
going into a debug exception loop on an instruction-breakpoint. See Section 17.3.1.1, “Instruction-Breakpoint 
Exception Condition,” for more information on the use of this flag. 


As noted in Section 6.8.3, execution of the MOV or POP instruction to load the SS register suppresses any instruc- 
tion breakpoint on the next instruction (just as if EFLAGS.RF were 1). 


2. The effect of the IOPL on these instructions is modified slightly when the virtual mode extension is enabled by setting the VME flag 
in control register CR4: see Section 19.3, “Interrupt and Exception Handling in Virtual-8086 Mode.” Behavior is also impacted by the 
PVI flag: see Section 19.4, “Protected-Mode Virtual Interrupts.” 


3. Nonmaskable interrupts and system-management interrupts may also be inhibited on the instruction boundary following such an 
execution of STI. 
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6.8.3 Masking Exceptions and Interrupts When Switching Stacks 


To switch to a different stack segment, software often uses a pair of instructions, for example: 


MOV SS, AX 
MOV ESP, StackTop 


(Software might also use the POP instruction to load SS and ESP.) 


If an interrupt or exception occurs after the new SS segment descriptor has been loaded but before the ESP register 
has been loaded, these two parts of the logical address into the stack space are inconsistent for the duration of the 
interrupt or exception handler (assuming that delivery of the interrupt or exception does not itself load a new stack 
pointer). 


To account for this situation, the processor prevents certain events from being delivered after execution of a MOV 
to SS instruction or a POP to SS instruction. The following items provide details: 


¢ Any instruction breakpoint on the next instruction is suppressed (as if EFLAGS.RF were 1). 


¢ Any data breakpoint on the MOV to SS instruction or POP to SS instruction is inhibited until the instruction 
boundary following the next instruction. 


¢ Any single-step trap that would be delivered following the MOV to SS instruction or POP to SS instruction 
(because EFLAGS.TF is 1) is suppressed. 


¢ The suppression and inhibition ends after delivery of an exception or the execution of the next instruction. 


¢ If a sequence of consecutive instructions each loads the SS register (using MOV or POP), only the first is 
guaranteed to inhibit or suppress events in this way. 


Intel recommends that software use the LSS instruction to load the SS register and ESP together. The problem 
identified earlier does not apply to LSS, and the LSS instruction does not inhibit events as detailed above. 


6.9 PRIORITIZATION OF CONCURRENT EVENTS 


If more than one event is pending at an instruction boundary (between execution of instructions), the processor 
services them in a predictable order. Table 6-2 shows the priority among classes of event sources. 


Table 6-2. Priority Among Concurrent Events 
Priority Description 


1 (Highest) Hardware Reset and Machine Checks 
- RESET 
- Machine Check (#MC) 


2 Trap on Task Switch 
- T flag in TSS is set (HDB) 


3 External Hardware Interventions 
- FLUSH 

- STOPCLK 

- SMI 

- INIT 


4 Traps on the Previous Instruction 
- Trap-class Debug Exceptions (#DB due to TF flag set or data/I-O breakpoint) 


Nonmaskable Interrupts (NMI) ? 


Maskable Hardware Interrupts 


Fault-class Debug Exceptions (#DB due to instruction breakpoint) 
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Table 6-2. Priority Among Concurrent Events (Contd.) 


Priority Description 

8 Faults from Fetching Next Instruction 
- Code-Segment Limit Violation (#GP) 
- Code Page Fault (#PF) 


- Control protection exception due to missing ENDBRANCH at target of an indirect call or jump (#CP) 


9 (Lowest) Faults from Decoding the Next Instruction 
- Instruction length > 15 bytes (#GP) 

- Invalid Opcode (#UD) 

- Coprocessor Not Available (#NM) 


NOTE 
1. The Intel® 486 processor and earlier processors group nonmaskable and maskable interrupts in the same priority class. 


The processor first services a pending event from the class which has the highest priority, transferring execution to 
the first instruction of the handler. Lower priority exceptions are discarded; lower priority interrupts are held 
pending. Discarded exceptions may be re-generated when the event handler returns execution to the point in the 
program or task where the original event occurred. While the priority among the classes listed in Table 6-2 is 
consistent across processor implementations, the priority of events within a class is implementation-dependent 
and may vary from processor to processor. 


Table 6-2 specifies the prioritization of events that may be pending at an instruction boundary. It does not specify 
the prioritization of faults that arise during instruction execution or event delivery (these include #BR, #TS, #NP, 

#SS, #GP, #PF, #AC, #MF, #XM, #VE, or #CP). It also does not apply to the events generated by the “Call to Inter- 
rupt Procedure” instructions (INT n, INTO, INT3, and INT1), as these events are integral to the execution of those 
instructions and do not occur between instructions. 


6.10 INTERRUPT DESCRIPTOR TABLE (IDT) 


The interrupt descriptor table (IDT) associates each exception or interrupt vector with a gate descriptor for the 
procedure or task used to service the associated exception or interrupt. Like the GDT and LDTs, the IDT is an array 
of 8-byte descriptors (in protected mode). Unlike the GDT, the first entry of the IDT may contain a descriptor. To 
form an index into the IDT, the processor scales the exception or interrupt vector by eight (the number of bytes in 
a gate descriptor). Because there are only 256 interrupt or exception vectors, the IDT need not contain more than 
256 descriptors. It can contain fewer than 256 descriptors, because descriptors are required only for the interrupt 
and exception vectors that may occur. All empty descriptor slots in the IDT should have the present flag for the 
descriptor set to 0. 


The base addresses of the IDT should be aligned on an 8-byte boundary to maximize performance of cache line 
fills. The limit value is expressed in bytes and is added to the base address to get the address of the last valid byte. 
A limit value of 0 results in exactly 1 valid byte. Because IDT entries are always eight bytes long, the limit should 
always be one less than an integral multiple of eight (that is, 8N - 1). 


The IDT may reside anywhere in the linear address space. As shown in Figure 6-1, the processor locates the IDT 
using the IDTR register. This register holds both a 32-bit base address and 16-bit limit for the IDT. 


The LIDT (load IDT register) and SIDT (store IDT register) instructions load and store the contents of the IDTR 
register, respectively. The LIDT instruction loads the IDTR register with the base address and limit held in a 
memory operand. This instruction can be executed only when the CPL is 0. It normally is used by the initialization 
code of an operating system when creating an IDT. An operating system also may use it to change from one IDT to 
another. The SIDT instruction copies the base and limit value stored in IDTR to memory. This instruction can be 
executed at any privilege level. 


If a vector references a descriptor beyond the limit of the IDT, a general-protection exception (#GP) is generated. 
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NOTE 


Because interrupts are delivered to the processor core only once, an incorrectly configured IDT 
could result in incomplete interrupt handling and/or the blocking of interrupt delivery. 

IA-32 architecture rules need to be followed for setting up IDTR base/limit/access fields and each 
field in the gate descriptors. The same apply for the Intel 64 architecture. This includes implicit 
referencing of the destination code segment through the GDT or LDT and accessing the stack. 


IDTR Register 
47 16 15 0 


IDT Base Address IDT Limit 


| Interrupt 


Descriptor Table (IDT 
>(F) p (IDT) 


Gate for 

Interrupt #n (n-1)*8 
Gate for 

Interrupt #3 16 
Gate for 

Interrupt #2 8 
Gate for 

{_____ Interrupt #1 0 
31 0 


Figure 6-1. Relationship of the IDTR and IDT 


6.11 IDT DESCRIPTORS 


The IDT may contain any of three kinds of gate descriptors: 
° Task-gate descriptor 

¢ Interrupt-gate descriptor 

* Trap-gate descriptor 


Figure 6-2 shows the formats for the task-gate, interrupt-gate, and trap-gate descriptors. The format of a task 
gate used in an IDT is the same as that of a task gate used in the GDT or an LDT (see Section 7.2.5, “Task-Gate 
Descriptor”). The task gate contains the segment selector for a TSS for an exception and/or interrupt handler task. 


Interrupt and trap gates are very similar to call gates (see Section 5.8.3, “Call Gates”). They contain a far pointer 
(segment selector and offset) that the processor uses to transfer program execution to a handler procedure in an 
exception- or interrupt-handler code segment. These gates differ in the way the processor handles the IF flag in the 
EFLAGS register (see Section 6.12.1.3, “Flag Usage By Exception- or Interrupt-Handler Procedure”). 
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Figure 6-2. IDT Gate Descriptors 


Task Gate 
31 16 15 14 13 12 8 7 
D 
P|} Pp |o0010 1 
L 
31 16 15 
TSS Segment Selector 
Interrupt Gate 
31 16 15 14 13 12 8 7 5 4 
D 
Offset 31..16 P|} Pp /0D110/0 00 
L 
31 16 15 
Segment Selector Offset 15..0 
Trap Gate 
31 16 15 14 13 12 8 7 5 4 
D 
Offset 31..16 P|} P j/0D111/0 00 
L 
31 16 15 
Segment Selector Offset 15..0 
DPL Descriptor Privilege Level 
Offset Offset to procedure entry point 
P Segment Present flag 
Selector Segment Selector for destination code segment 
D Size of gate: 1 = 32 bits; 0 = 16 bits 
Reserved 


INTERRUPT AND EXCEPTION HANDLING 


The processor handles calls to exception- and interrupt-handlers similar to the way it handles calls with a CALL 
instruction to a procedure or a task. When responding to an exception or interrupt, the processor uses the excep- 
tion or interrupt vector as an index to a descriptor in the IDT. If the index points to an interrupt gate or trap gate, 
the processor calls the exception or interrupt handler in a manner similar to a CALL to a call gate (see Section 
5.8.2, “Gate Descriptors,” through Section 5.8.6, “Returning from a Called Procedure”). If index points to a task 
gate, the processor executes a task switch to the exception- or interrupt-handler task in a manner similar to a CALL 
to a task gate (see Section 7.3, “Task Switching”). 
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6.12.1 €xception- or Interrupt-Handler Procedures 


An interrupt gate or trap gate references an exception- or interrupt-handler procedure that runs in the context of 
the currently executing task (see Figure 6-3). The segment selector for the gate points to a segment descriptor for 
an executable code segment in either the GDT or the current LDT. The offset field of the gate descriptor points to 

the beginning of the exception- or interrupt-handling procedure. 


Destination 
IDT Code Segment 
Interrupt 
Offset Procedure 
Interrupt Interrupt or ala cel 
Vector Trap Gate 
> 
Segment Selector 
GDT or LDT 
Base 
Address 
j Segment 
Descriptor 


Figure 6-3. Interrupt Procedure Call 
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When the processor performs a call to the exception- or interrupt-handler procedure: 


If the handler procedure is going to be executed at a numerically lower privilege level, a stack switch occurs. 
When the stack switch occurs: 


a. The segment selector and stack pointer for the stack to be used by the handler are obtained from the TSS 
for the currently executing task. On this new stack, the processor pushes the stack segment selector and 
stack pointer of the interrupted procedure. 


b. The processor then saves the current state of the EFLAGS, CS, and EIP registers on the new stack (see 
Figure 6-4). 


c. If an exception causes an error code to be saved, it is pushed on the new stack after the EIP value. 
If the handler procedure is going to be executed at the same privilege level as the interrupted procedure: 


a. The processor saves the current state of the EFLAGS, CS, and EIP registers on the current stack (see Figure 
6-4). 


b. If an exception causes an error code to be saved, it is pushed on the current stack after the EIP value. 


Stack Usage with No 
Privilege-Level Change 
Interrupted Procedure’s 
and Handler’s Stack 


~x—— ESP Before 
EFLAGS Transfer to Handler 
CS 
EIP 
ErrorCode «ESP After 
Transfer to Handler 


Stack Usage with 
Privilege-Level Change 


Interrupted Procedure’s 


Stack Handler’s Stack 
+<——\ESP Before 

Transfer to Handler ss 

ESP 
EFLAGS 
cS 
EIP 
ESP After——»|__ Error Code 
Transfer to Handler 


Figure 6-4. Stack Usage on Transfers to Interrupt and Exception-Handling Routines 


To return from an exception- or interrupt-handler procedure, the handler must use the IRET (or IRETD) instruction. 
The IRET instruction is similar to the RET instruction except that it restores the saved flags into the EFLAGS 
register. The IOPL field of the EFLAGS register is restored only if the CPLis 0. The IF flag is changed only if the CPL 
is less than or equal to the IOPL. See Chapter 3, “Instruction Set Reference, A-L,” of the Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 2A, for a description of the complete operation performed by the 
IRET instruction. 


If a stack switch occurred when calling the handler procedure, the IRET instruction switches back to the interrupted 
procedure’s stack on the return. 


Vol. 3A 6-13 


INTERRUPT AND EXCEPTION HANDLING 


6.12.1.1 Shadow Stack Usage on Transfers to Interrupt and Exception Handling Routines 


When the processor performs a call to the exception- or interrupt-handler procedure: 


If the handler procedure is going to be execute at a numerically lower privilege level, a shadow stack switch 
occurs. When the shadow stack switch occurs: 


a. 


On a transfer from privilege level 3, if shadow stacks are enabled at privilege level 3 then the SSP is saved 
to the IA32_PL3_SSP MSR. 


If shadow stacks are enabled at the privilege level where the handler will execute then the shadow stack for 
the handler is obtained from one of the following MSRs based on the privilege level at which the handler 
executes. 


¢ JA32_PL2_SSP if handler executes at privilege level 2. 
¢ JA32_PL1_SSP if handler executes at privilege level 1. 
¢ JA32_PLO_SSP if handler executes at privilege level 0. 


The SSP obtained is then verified to ensure it points to a valid supervisory shadow stack that is not currently 
active by verifying a supervisor shadow stack token at the address pointed to by the SSP. The operations 
performed to verify and acquire the supervisor shadow stack token by making it busy are as described in 
Section 18.2.3 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


On this new shadow stack, the processor pushes the CS, LIP (CS.base + EIP), and SSP of the interrupted 
procedure if the interrupted procedure was executing at privilege level less than 3; see Figure 6-5. 


If the handler procedure is going to be executed at the same privilege level as the interrupted procedure and 
shadow stacks are enabled at current privilege level: 


a. 


The processor saves the current state of the CS, LIP (CS.base + EIP), and SSP registers on the current 
shadow stack; see Figure 6-5. 
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Shadow Stack Usage with 
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Shadow Stack Usage with Privilege- 
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Interrupted Procedure’s Handler's Shadow Stack 
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Shadow Stack Usage with Privilege- 
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LIP 
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Figure 6-5. Shadow Stack Usage on Transfers to Interrupt and Exception-Handling Routines 


To return from an exception- or interrupt-handler procedure, the handler must use the IRET (or IRETD) instruction. 


When executing a return from an interrupt or exception handler from the same privilege level as the interrupted 
procedure, the processor performs these actions to enforce return address protection: 


¢ Restores the CS and EIP registers to their values prior to the interrupt or exception. 


Vol. 3A 6-15 


INTERRUPT AND EXCEPTION HANDLING 


If shadow stack is enabled: 


— Compares the values on shadow stack at address SSP+8 (the LIP) and SSP+16 (the CS) to the CS and 
(CS.base + EIP) popped from the stack and causes a control protection exception (#CP(FAR-RET/IRET)) if 
they do not match. 


— Pops the top-of-stack value (the SSP prior to the interrupt or exception) from shadow stack into SSP 
register. 


When executing a return from an interrupt or exception handler from a different privilege level than the interrupted 
procedure, the processor performs the actions below. 


If shadow stack is enabled at current privilege level: 
— If SSP is not aligned to 8 bytes then causes a control protection exception (#CP(FAR-RET/IRET)). 
— If privilege level of the procedure being returned to is less than 3 (returning to supervisor mode): 


¢ Compares the values on shadow stack at address SSP+8 (the LIP) and SSP+16 (the CS) to the CS and 
(CS.base + EIP) popped from the stack and causes a control protection exception (#CP(FAR-RET/IRET)) 
if they do not match. 


* Temporarily saves the top-of-stack value (the SSP of the procedure being returned to) internally. 


— Ifa busy supervisor shadow stack token is present at address SSP+24, then marks the token free using 
operations described in section Section 18.2.3 of the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1. 


— Ifthe privilege level of the procedure being returned to is less than 3 (returning to supervisor mode), 
restores the SSP register from the internally saved value. 


— Ifthe privilege level of the procedure being returned to is 3 (returning to user mode) and shadow stack is 
enabled at privilege level 3, then restores the SSP register with value of IA32_PL3_SSP MSR. 


6.12.1.2 Protection of Exception- and Interrupt-Handler Procedures 


The privilege-level protection for exception- and interrupt-handler procedures is similar to that used for ordinary 
procedure calls when called through a call gate (see Section 5.8.4, “Accessing a Code Segment Through a Call 
Gate”). The processor does not permit transfer of execution to an exception- or interrupt-handler procedure in a 
less privileged code segment (numerically greater privilege level) than the CPL. 


An attempt to violate this rule results in a general-protection exception (#GP). The protection mechanism for 
exception- and interrupt-handler procedures is different in the following ways: 


Because interrupt and exception vectors have no RPL, the RPL is not checked on implicit calls to exception and 
interrupt handlers. 


The processor checks the DPL of the interrupt or trap gate only if an exception or interrupt is generated with an 
INT n, INT3, or INTO instruction.* Here, the CPL must be less than or equal to the DPL of the gate. This 
restriction prevents application programs or procedures running at privilege level 3 from using a software 
interrupt to access critical exception handlers, such as the page-fault handler, providing that those handlers are 
placed in more privileged code segments (numerically lower privilege level). For hardware-generated interrupts 
and processor-detected exceptions, the processor ignores the DPL of interrupt and trap gates. 


Because exceptions and interrupts generally do not occur at predictable times, these privilege rules effectively 
impose restrictions on the privilege levels at which exception and interrupt- handling procedures can run. Either of 
the following techniques can be used to avoid privilege-level violations. 


The exception or interrupt handler can be placed in a conforming code segment. This technique can be used for 
handlers that only need to access data available on the stack (for example, divide error exceptions). If the 
handler needs data from a data segment, the data segment needs to be accessible from privilege level 3, which 
would make it unprotected. 


The handler can be placed in a nonconforming code segment with privilege level 0. This handler would always 
run, regardless of the CPL that the interrupted program or task is running at. 


This check is not performed by execution of the INT1 instruction (opcode F1); it would be performed by execution of INT 1 (opcode 
CD 01). 
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6.12.1.3 Flag Usage By Exception- or Interrupt-Handler Procedure 


When accessing an exception or interrupt handler through either an interrupt gate or a trap gate, the processor 

clears the TF flag in the EFLAGS register after it saves the contents of the EFLAGS register on the stack. (On calls 
to exception and interrupt handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register, after 
they are saved on the stack.) Clearing the TF flag prevents instruction tracing from affecting interrupt response and 
ensures that no single-step exception will be delivered after delivery to the handler. A subsequent IRET instruction 
restores the TF (and VM, RF, and NT) flags to the values in the saved contents of the EFLAGS register on the stack. 


The only difference between an interrupt gate and a trap gate is the way the processor handles the IF flag in the 
EFLAGS register. When accessing an exception- or interrupt-handling procedure through an interrupt gate, the 
processor clears the IF flag to prevent other interrupts from interfering with the current interrupt handler. A subse- 
quent IRET instruction restores the IF flag to its value in the saved contents of the EFLAGS register on the stack. 
Accessing a handler procedure through a trap gate does not affect the IF flag. 


6.12.2 Interrupt Tasks 


When an exception or interrupt handler is accessed through a task gate in the IDT, a task switch results. Handling 
an exception or interrupt with a separate task offers several advantages: 


¢ The entire context of the interrupted program or task is saved automatically. 


¢ Anew TSS permits the handler to use a new privilege level 0 stack when handling the exception or interrupt. If 
an exception or interrupt occurs when the current privilege level 0 stack is corrupted, accessing the handler 
through a task gate can prevent a system crash by providing the handler with a new privilege level 0 stack. 


¢ The handler can be further isolated from other tasks by giving it a separate address space. This is done by 
giving it a separate LDT. 


The disadvantage of handling an interrupt with a separate task is that the amount of machine state that must be 
saved on a task switch makes it slower than using an interrupt gate, resulting in increased interrupt latency. 


A task gate in the IDT references a TSS descriptor in the GDT (see Figure 6-6). A switch to the handler task is 
handled in the same manner as an ordinary task switch (see Section 7.3, “Task Switching”). The link back to the 
interrupted task is stored in the previous task link field of the handler task’s TSS. If an exception caused an error 
code to be generated, this error code is copied to the stack of the new task. 


When exception- or interrupt-handler tasks are used in an operating system, there are actually two mechanisms 
that can be used to dispatch tasks: the software scheduler (part of the operating system) and the hardware sched- 
uler (part of the processor's interrupt mechanism). The software scheduler needs to accommodate interrupt tasks 
that may be dispatched when interrupts are enabled. 


Vol. 3A 6-17 


INTERRUPT AND EXCEPTION HANDLING 


6.13 


NOTE 


Because IA-32 architecture tasks are not re-entrant, an interrupt-handler task must disable 
interrupts between the time it completes handling the interrupt and the time it executes the IRET 
instruction. This action prevents another interrupt from occurring while the interrupt task’s TSS is 
still marked busy, which would cause a general-protection (#GP) exception. 


TSS for Interrupt- 
IDT Handling Task 


Interrupt 


Vector Task Gate 


TSS Selector TSS 
Base 
GDT Address 


— TSS Descriptor 


Figure 6-6. Interrupt Task Switch 


ERROR CODE 


When an exception condition is related to a specific segment selector or IDT vector, the processor pushes an error 
code onto the stack of the exception handler (whether it is a procedure or task). The error code has the format 
shown in Figure 6-7. The error code resembles a segment selector; however, instead of a TI flag and RPL field, the 
error code contains 3 flags: 


EXT 


IDT 


TI 


External event (bit 0) — When set, indicates that the exception occurred during delivery of an 
event external to the program, such as an interrupt or an earlier exception.” The bit is cleared if the 
exception occurred during delivery of a software interrupt (INT n, INT3, or INTO). 


Descriptor location (bit 1) — When set, indicates that the index portion of the error code refers 
to a gate descriptor in the IDT; when clear, indicates that the index refers to a descriptor in the GDT 
or the current LDT. 


GDT/LDT (bit 2) — Only used when the IDT flag is clear. When set, the TI flag indicates that the 
index portion of the error code refers to a segment or gate descriptor in the LDT; when clear, it indi- 
cates that the index refers to a descriptor in the current GDT. 


5. The bit is also set if the exception occurred during delivery of INT1. 
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Reserved Segment Selector Index 


Figure 6-7. Error Code 


The segment selector index field provides an index into the IDT, GDT, or current LDT to the segment or gate 
selector being referenced by the error code. In some cases the error code is null (all bits are clear except possibly 
EXT). A null error code indicates that the error was not caused by a reference to a specific segment or that a null 
segment selector was referenced in an operation. 


The format of the error code is different for page-fault exceptions (#PF). See the “Interrupt 14—Page-Fault Excep- 
tion (#PF)” section in this chapter. 


The format of the error code is different for control protection exceptions (#CP). See the “Interrupt 21—Control 
Protection Exception (#CP)” section in this chapter. 


The error code is pushed on the stack as a doubleword or word (depending on the default interrupt, trap, or task 
gate size). To keep the stack aligned for doubleword pushes, the upper half of the error code is reserved. Note that 
the error code is not popped when the IRET instruction is executed to return from an exception handler, so the 
handler must remove the error code before executing a return. 


Error codes are not pushed on the stack for exceptions that are generated externally (with the INTR or LINT[1:0] 
pins) or the INT rn instruction, even if an error code is normally produced for those exceptions. 


6.14 EXCEPTION AND INTERRUPT HANDLING IN 64-BIT MODE 


In 64-bit mode, interrupt and exception handling is similar to what has been described for non-64-bit modes. The 
following are the exceptions: 


¢ All interrupt handlers pointed by the IDT are in 64-bit code (this does not apply to the SMI handler). 
° The size of interrupt-stack pushes is fixed at 64 bits; and the processor uses 8-byte, zero extended stores. 


¢ The stack pointer (SS:RSP) is pushed unconditionally on interrupts. In legacy modes, this push is conditional 
and based on a change in current privilege level (CPL). 


¢ The new SS is set to NULL if there is a change in CPL. 

¢ _IRET behavior changes. 

¢ There is a new interrupt stack-switch mechanism and a new interrupt shadow stack-switch mechanism. 
¢ The alignment of interrupt stack frame is different. 


6.14.1 64-Bit Mode IDT 


Interrupt and trap gates are 16 bytes in length to provide a 64-bit offset for the instruction pointer (RIP). The 64- 
bit RIP referenced by interrupt-gate descriptors allows an interrupt service routine to be located anywhere in the 
linear-address space. See Figure 6-8. 
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Figure 6-8. 64-Bit IDT Gate Descriptors 


In 64-bit mode, the IDT index is formed by scaling the interrupt vector by 16. The first eight bytes (bytes 7:0) of a 
64-bit mode interrupt gate are similar but not identical to legacy 32-bit interrupt gates. The type field (bits 11:8 in 
bytes 7:4) is described in Table 3-2. The Interrupt Stack Table (IST) field (bits 4:0 in bytes 7:4) is used by the stack 
switching mechanisms described in Section 6.14.5, “Interrupt Stack Table.” Bytes 11:8 hold the upper 32 bits of 
the target RIP (interrupt segment offset) in canonical form. A general-protection exception (#GP) is generated if 
software attempts to reference an interrupt gate with a target RIP that is not in canonical form. 


The target code segment referenced by the interrupt gate must be a 64-bit code segment (CS.L = 1, CS.D = 0). If 
the target is not a 64-bit code segment, a general-protection exception (#GP) is generated with the IDT vector 
number reported as the error code. 


Only 64-bit interrupt and trap gates can be referenced in IA-32e mode (64-bit mode and compatibility mode). 
Legacy 32-bit interrupt or trap gate types (OEH or OFH) are redefined in IA-32e mode as 64-bit interrupt and trap 
gate types. No 32-bit interrupt or trap gate type exists in IA-32e mode. If a reference is made to a 16-bit interrupt 
or trap gate (06H or 07H), a general-protection exception (#GP(0)) is generated. 


6.14.2 64-Bit Mode Stack Frame 


In legacy mode, the size of an IDT entry (16 bits or 32 bits) determines the size of interrupt-stack-frame pushes. 
SS:ESP is pushed only on a CPL change. In 64-bit mode, the size of interrupt stack-frame pushes is fixed at eight 
bytes. This is because only 64-bit mode gates can be referenced. 64-bit mode also pushes SS:RSP unconditionally, 
rather than only on a CPL change. 


When shadow stacks are enabled at the interrupt handler’s privilege level and the interrupted procedure was not 
executing at a privilege level 3, then the processor pushes the CS:LIP:SSP of the interrupted procedure on the 
shadow stack of the interrupt handler (where LIP is the linear address of the return address). 


Aside from error codes, pushing SS:RSP unconditionally presents operating systems with a consistent interrupt- 
stackframe size across all interrupts. Interrupt service-routine entry points that handle interrupts generated by the 
INTn instruction or external INTR# signal can push an additional error code place-holder to maintain consistency. 


In legacy mode, the stack pointer may be at any alignment when an interrupt or exception causes a stack frame to 
be pushed. This causes the stack frame and succeeding pushes done by an interrupt handler to be at arbitrary 
alignments. In IA-32e mode, the RSP is aligned to a 16-byte boundary before pushing the stack frame. The stack 
frame itself is aligned on a 16-byte boundary when the interrupt handler is called. The processor can arbitrarily 
realign the new RSP on interrupts because the previous (possibly unaligned) RSP is unconditionally saved on the 
newly aligned stack. The previous RSP will be automatically restored by a subsequent IRET. 
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Aligning the stack permits exception and interrupt frames to be aligned on a 16-byte boundary before interrupts 
are re-enabled. This allows the stack to be formatted for optimal storage of 16-byte XMM registers, which enables 
the interrupt handler to use faster 16-byte aligned loads and stores (MOVAPS rather than MOVUPS) to save and 
restore XMM registers. 


Although the RSP alignment is always performed when LMA = 1, it is only of consequence for the kernel-mode case 
where there is no stack switch or IST used. For a stack switch or IST, the OS would have presumably put suitably 
aligned RSP values in the TSS. 


6.14.3. IRET in lA-32e Mode 


In IA-32e mode, IRET executes with an 8-byte operand size. There is nothing that forces this requirement. The 
stack is formatted in such a way that for actions where IRET is required, the 8-byte IRET operand size works 
correctly. 


Because interrupt stack-frame pushes are always eight bytes in IA-32e mode, an IRET must pop eight byte items 
off the stack. This is accomplished by preceding the IRET with a 64-bit operand-size prefix. The size of the pop is 
determined by the address size of the instruction. The SS/ESP/RSP size adjustment is determined by the stack 
size. 


IRET pops SS:RSP unconditionally off the interrupt stack frame only when it is executed in 64-bit mode. In compat- 
ibility mode, IRET pops SS:RSP off the stack only if there is a CPL change. This allows legacy applications to 
execute properly in compatibility mode when using the IRET instruction. 64-bit interrupt service routines that exit 
with an IRET unconditionally pop SS:RSP off of the interrupt stack frame, even if the target code segment is 
running in 64-bit mode or at CPL = O. This is because the original interrupt always pushes SS:RSP. 


When shadow stacks are enabled and the target privilege level is not 3, the CS:LIP from the shadow stack frame is 
compared to the return linear address formed by CS:EIP from the stack. If they do not match then the processor 
caused a control protection exception (#CP(FAR-RET/IRET)), else the processor pops the SSP of the interrupted 
procedure from the shadow stack. If the target privilege level is 3 and shadow stacks are enabled at privilege level 
3, then the SSP for the interrupted procedure is restored from the IA32_PL3_SSP MSR. 


In IA-32e mode, IRET is allowed to load a NULL SS under certain conditions. If the target mode is 64-bit mode and 
the target CPL + 3, IRET allows SS to be loaded with a NULL selector. As part of the stack switch mechanism, an 
interrupt or exception sets the new SS to NULL, instead of fetching a new SS selector from the TSS and loading the 
corresponding descriptor from the GDT or LDT. The new SS selector is set to NULL in order to properly handle 
returns from subsequent nested far transfers. If the called procedure itself is interrupted, the NULL SS is pushed 
on the stack frame. On the subsequent IRET, the NULL SS on the stack acts as a flag to tell the processor not to 
load a new SS descriptor. 


6.14.4 Stack Switching in IA-32e Mode 


The IA-32 architecture provides a mechanism to automatically switch stack frames in response to an interrupt. The 
64-bit extensions of Intel 64 architecture implement a modified version of the legacy stack-switching mechanism 
and an alternative stack-switching mechanism called the interrupt stack table (IST). 


In IA-32 modes, the legacy IA-32 stack-switch mechanism is unchanged. In IA-32e mode, the legacy stack-switch 
mechanism is modified. When stacks are switched as part of a 64-bit mode privilege-level change (resulting from 
an interrupt), a new SS descriptor is not loaded. IA-32e mode loads only an inner-level RSP from the TSS. The new 
SS selector is forced to NULL and the SS selector’s RPL field is set to the new CPL. The new SS is set to NULL in 
order to handle nested far transfers (far CALL, INT, interrupts and exceptions). The old SS and RSP are saved on 
the new stack (Figure 6-9). On the subsequent IRET, the old SS is popped from the stack and loaded into the SS 
register. 


In summary, a stack switch in IA-32e mode works like the legacy stack switch, except that a new SS selector is not 
loaded from the TSS. Instead, the new SS is forced to NULL. 
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Figure 6-9. IA-32e Mode Stack Usage After Privilege Level Change 


6.14.5 Interrupt Stack Table 


In IA-32e mode, a new interrupt stack table (IST) mechanism is available as an alternative to the modified legacy 
stack-switching mechanism described above. This mechanism unconditionally switches stacks when it is enabled. 

It can be enabled on an individual interrupt-vector basis using a field in the IDT entry. This means that some inter- 
rupt vectors can use the modified legacy mechanism and others can use the IST mechanism. 


The IST mechanism is only available in IA-32e mode. It is part of the 64-bit mode TSS. The motivation for the IST 
mechanism is to provide a method for specific interrupts (such as NMI, double-fault, and machine-check) to always 
execute on a known good stack. In legacy mode, interrupts can use the task-switch mechanism to set up a known- 
good stack by accessing the interrupt service routine through a task gate located in the IDT. However, the legacy 
task-switch mechanism is not supported in IA-32e mode. 


The IST mechanism provides up to seven IST pointers in the TSS. The pointers are referenced by an interrupt-gate 
descriptor in the interrupt-descriptor table (IDT); see Figure 6-8. The gate descriptor contains a 3-bit IST index 
field that provides an offset into the IST section of the TSS. Using the IST mechanism, the processor loads the value 
pointed by an IST pointer into the RSP. 


When an interrupt occurs, the new SS selector is forced to NULL and the SS selector’s RPL field is set to the new 
CPL. The old SS, RSP, RFLAGS, CS, and RIP are pushed onto the new stack. Interrupt processing then proceeds as 
normal. If the IST index is zero, the modified legacy stack-switching mechanism described above is used. 


To support this stack-switching mechanism with shadow stacks enabled, the processor provides an MSR, 
IA32_INTERRUPT_SSP_TABLE, to program the linear address of a table of seven shadow stack pointers that are 
selected using the IST index from the gate descriptor. To switch to a shadow stack selected from the interrupt 
shadow stack table pointed to by the IA32_INTERRUPT_SSP_TABLE, the processor requires that the shadow stack 
addresses programmed into this table point to a supervisor shadow stack token; see Figure 6-10. 
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Figure 6-10. Interrupt Shadow Stack Table 


6.15 EXCEPTION AND INTERRUPT REFERENCE 


The following sections describe conditions which generate exceptions and interrupts. They are arranged in the 
order of vector numbers. The information contained in these sections are as follows: 


Exception Class — Indicates whether the exception class is a fault, trap, or abort type. Some exceptions can 
be either a fault or trap type, depending on when the error condition is detected. (This section is not applicable 
to interrupts.) 


Description — Gives a general description of the purpose of the exception or interrupt type. It also describes 
how the processor handles the exception or interrupt. 


Exception Error Code — Indicates whether an error code is saved for the exception. If one is saved, the 
contents of the error code are described. (This section is not applicable to interrupts.) 


Saved Instruction Pointer — Describes which instruction the saved (or return) instruction pointer points to. 
It also indicates whether the pointer can be used to restart a faulting instruction. 


Program State Change — Describes the effects of the exception or interrupt on the state of the currently 
running program or task and the possibilities of restarting the program or task without loss of continuity. 
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Interrupt O—Divide Error Exception (#DE) 


Exception Class Fault. 


Description 


Indicates the divisor operand for a DIV or IDIV instruction is 0 or that the result cannot be represented in the 
number of bits specified for the destination operand. 


Exception Error Code 
None. 


Saved Instruction Pointer 


Saved contents of CS and EIP registers point to the instruction that generated the exception. 


Program State Change 


A program-state change does not accompany the divide error, because the exception occurs before the faulting 
instruction is executed. 
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Interrupt 1—Debug Exception (#DB) 


Exception Class Trap or Fault. The exception handler can distinguish between traps or faults by exam- 
ining the contents of DR6 and the other debug registers. 


Description 


Indicates that one or more of several debug-exception conditions has been detected. Whether the exception is a 
fault or a trap depends on the condition (see Table 6-3). See Chapter 17, “Debug, Branch Profile, TSC, and Intel® 
Resource Director Technology (Intel® RDT) Features,” for detailed information about the debug exceptions. 


Table 6-3. Debug Exception Conditions and Corresponding Exception Classes 


Exception Condition Exception Class 
Instruction fetch breakpoint Fault 

Data read or write breakpoint Trap 

I/O read or write breakpoint Trap 

General detect condition (in conjunction with in-circuit emulation) Fault 
Single-step Trap 
Task-switch Trap 

Execution of INT1! Trap 
NOTES: 


1. Hardware vendors may use the INT1 instruction for hardware debug. For that reason, Intel recommends software vendors instead 
use the INT3 instruction for software breakpoints. 


Exception Error Code 
None. An exception handler can examine the debug registers to determine which condition caused the exception. 


Saved Instruction Pointer 
Fault — Saved contents of CS and EIP registers point to the instruction that generated the exception. 


Trap — Saved contents of CS and EIP registers point to the instruction following the instruction that generated the 
exception. 


Program State Change 


Fault — A program-state change does not accompany the debug exception, because the exception occurs before 
the faulting instruction is executed. The program can resume normal execution upon returning from the debug 
exception handler. 


Trap — A program-state change does accompany the debug exception, because the instruction or task switch being 
executed is allowed to complete before the exception is generated. However, the new state of the program is not 
corrupted and execution of the program can continue reliably. 


The following items detail the treatment of debug exceptions on the instruction boundary following execution of the 
MOV or the POP instruction that loads the SS register: 


¢ If EFLAGS.TF is 1, no single-step trap is generated. 


¢ If the instruction encounters a data breakpoint, the resulting debug exception is delivered after completion of 
the instruction after the MOV or POP. This occurs even if the next instruction is INT n, INT3, or INTO. 


e = Any instruction breakpoint on the instruction after the MOV or POP is suppressed (as if EFLAGS.RF were 1). 


Any debug exception inside an RTM region causes a transactional abort and, by default, redirects control flow to the 
fallback instruction address. If advanced debugging of RTM transactional regions has been enabled, any transac- 
tional abort due to a debug exception instead causes execution to roll back to just before the XBEGIN instruction 
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and then delivers a #DB. See Section 16.3.7, “RTM-Enabled Debugger Support,” of Inte/® 64 and IA-32 Architec- 
tures Software Developer’s Manual, Volume 1. 
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Interrupt 2—NMI Interrupt 


Exception Class Not applicable. 


Description 


The nonmaskable interrupt (NMI) is generated externally by asserting the processor’s NMI pin or through an NMI 
request set by the I/O APIC to the local APIC. This interrupt causes the NMI interrupt handler to be called. 


Exception Error Code 
Not applicable. 


Saved Instruction Pointer 


The processor always takes an NMI interrupt on an instruction boundary. The saved contents of CS and EIP regis- 
ters point to the next instruction to be executed at the point the interrupt is taken. See Section 6.5, “Exception 
Classifications,” for more information about when the processor takes NMI interrupts. 


Program State Change 


The instruction executing when an NMI interrupt is received is completed before the NMI is generated. A program 
or task can thus be restarted upon returning from an interrupt handler without loss of continuity, provided the 
interrupt handler saves the state of the processor before handling the interrupt and restores the processor’s state 
prior to a return. 
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Interrupt 3—Breakpoint Exception (#BP) 


Exception Class Trap. 


Description 


Indicates that a breakpoint instruction (INT3, opcode CC) was executed, causing a breakpoint trap to be gener- 
ated. Typically, a debugger sets a breakpoint by replacing the first opcode byte of an instruction with the opcode for 
the INT3 instruction. (The INT3 instruction is one byte long, which makes it easy to replace an opcode in a code 
segment in RAM with the breakpoint opcode.) The operating system or a debugging tool can use a data segment 
mapped to the same physical address space as the code segment to place an INT3 instruction in places where it is 
desired to call the debugger. 


With the P6 family, Pentium, Intel486, and Intel386 processors, it is more convenient to set breakpoints with the 
debug registers. (See Section 17.3.2, “Breakpoint Exception (#BP)—Interrupt Vector 3,” for information about the 
breakpoint exception.) If more breakpoints are needed beyond what the debug registers allow, the INT3 instruction 
can be used. 


Any breakpoint exception inside an RTM region causes a transactional abort and, by default, redirects control flow 
to the fallback instruction address. If advanced debugging of RTM transactional regions has been enabled, any 
transactional abort due to a break exception instead causes execution to roll back to just before the XBEGIN 
instruction and then delivers a debug exception (#DB) — nota breakpoint exception. See Section 16.3.7, “RTM- 
Enabled Debugger Support,” of Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


A breakpoint exception can also be generated by executing the INT n instruction with an operand of 3. The action 
of this instruction (INT 3) is slightly different than that of the INT3 instruction (see “INT n/INTO/INT3/INT1—Call to 
Interrupt Procedure” in Chapter 3 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
2A). 


Exception Error Code 
None. 


Saved Instruction Pointer 
Saved contents of CS and EIP registers point to the instruction following the INT3 instruction. 


Program State Change 


Even though the EIP points to the instruction following the breakpoint instruction, the state of the program is 
essentially unchanged because the INT3 instruction does not affect any register or memory locations. The 
debugger can thus resume the suspended program by replacing the INT3 instruction that caused the breakpoint 
with the original opcode and decrementing the saved contents of the EIP register. Upon returning from the 
debugger, program execution resumes with the replaced instruction. 
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Interrupt 4—Overflow Exception (#OF) 


Exception Class Trap. 


Description 


Indicates that an overflow trap occurred when an INTO instruction was executed. The INTO instruction checks the 
state of the OF flag in the EFLAGS register. If the OF flag is set, an overflow trap is generated. 


Some arithmetic instructions (such as the ADD and SUB) perform both signed and unsigned arithmetic. These 
instructions set the OF and CF flags in the EFLAGS register to indicate signed overflow and unsigned overflow, 
respectively. When performing arithmetic on signed operands, the OF flag can be tested directly or the INTO 
instruction can be used. The benefit of using the INTO instruction is that if the overflow exception is detected, an 
exception handler can be called automatically to handle the overflow condition. 


Exception Error Code 


None. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the instruction following the INTO instruction. 


Program State Change 


Even though the EIP points to the instruction following the INTO instruction, the state of the program is essentially 
unchanged because the INTO instruction does not affect any register or memory locations. The program can thus 
resume normal execution upon returning from the overflow exception handler. 
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Interrupt 5—BOUND Range Exceeded Exception (#BR) 


Exception Class Fault. 


Description 


Indicates that a BOUND-range-exceeded fault occurred when a BOUND instruction was executed. The BOUND 
instruction checks that a signed array index is within the upper and lower bounds of an array located in memory. If 
the array index is not within the bounds of the array, a BOUND-range-exceeded fault is generated. 


Exception Error Code 
None. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the BOUND instruction that generated the exception. 


Program State Change 


A program-state change does not accompany the bounds-check fault, because the operands for the BOUND 
instruction are not modified. Returning from the BOUND-range-exceeded exception handler causes the BOUND 
instruction to be restarted. 
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Interrupt 6—Invalid Opcode Exception (#UD) 


Exception Class Fault. 


Description 
Indicates that the processor did one of the following things: 
¢ Attempted to execute an invalid or reserved opcode. 


¢ Attempted to execute an instruction with an operand type that is invalid for its accompanying opcode; for 
example, the source operand for a LES instruction is not a memory location. 


e Attempted to execute an MMX or SSE/SSE2/SSE3 instruction on an Intel 64 or IA-32 processor that does not 
support the MMX technology or SSE/SSE2/SSE3/SSSE3 extensions, respectively. CPUID feature flags MMX (bit 
23), SSE (bit 25), SSE2 (bit 26), SSE3 (ECX, bit 0), SSSE3 (ECX, bit 9) indicate support for these extensions. 

e Attempted to execute an MMX instruction or SSE/SSE2/SSE3/SSSE3 SIMD instruction (with the exception of 
the MOVNTI, PAUSE, PREFETCHh, SFENCE, LFENCE, MFENCE, CLFLUSH, MONITOR, and MWAIT instructions) 
when the EM flag in control register CRO is set (1). 

¢ Attempted to execute an SSE/SE2/SSE3/SSSE3 instruction when the OSFXSR bit in control register CR4 is 
clear (0). Note this does not include the following SSE/SSE2/SSE3 instructions: MASKMOVQ, MOVNTQ, 
MOVNTI, PREFETCHA, SFENCE, LFENCE, MFENCE, and CLFLUSH; or the 64-bit versions of the PAVGB, PAVGW, 
PEXTRW, PINSRW, PMAXSW, PMAXUB, PMINSW, PMINUB, PMOVMSKB, PMULHUW, PSADBW, PSHUFW, PADDQ, 
PSUBQ, PALIGNR, PABSB, PABSD, PABSW, PHADDD, PHADDSW, PHADDW, PHSUBD, PHSUBSW, PHSUBW, 
PMADDUBSM, PMULHRSW, PSHUFB, PSIGNB, PSIGND, and PSIGNW. 

¢ Attempted to execute an SSE/SSE2/SSE3/SSSE3 instruction on an Intel 64 or IA-32 processor that caused a 
SIMD floating-point exception when the OSXMMEXCPT bit in control register CR4 is clear (0). 

e Executed a UDO, UD1 or UD2 instruction. Note that even though it is the execution of the UDO, UD1 or UD2 
instruction that causes the invalid opcode exception, the saved instruction pointer will still points at the UDO, 
UD1 or UD2 instruction. 

° Detected a LOCK prefix that precedes an instruction that may not be locked or one that may be locked but the 
destination operand is not a memory location. 

e Attempted to execute an LLDT, SLDT, LTR, STR, LSL, LAR, VERR, VERW, or ARPL instruction while in real- 
address or virtual-8086 mode. 


¢ Attempted to execute the RSM instruction when not in SMM mode. 


In Intel 64 and IA-32 processors that implement out-of-order execution microarchitectures, this exception is not 
generated until an attempt is made to retire the result of executing an invalid instruction; that is, decoding and 
speculatively attempting to execute an invalid opcode does not generate this exception. Likewise, in the Pentium 
processor and earlier IA-32 processors, this exception is not generated as the result of prefetching and preliminary 
decoding of an invalid instruction. (See Section 6.5, “Exception Classifications,” for general rules for taking of inter- 
rupts and exceptions.) 


The opcodes D6 and F1 are undefined opcodes reserved by the Intel 64 and IA-32 architectures. These opcodes, 
even though undefined, do not generate an invalid opcode exception. 


Exception Error Code 
None. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the instruction that generated the exception. 
Program State Change 


A program-state change does not accompany an invalid-opcode fault, because the invalid instruction is not 
executed. 
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Interrupt 7—Device Not Available Exception (#NM) 


Exception Class Fault. 


Description 
Indicates one of the following things: 
The device-not-available exception is generated by either of three conditions: 


¢ The processor executed an x87 FPU floating-point instruction while the EM flag in control register CRO was set 
(1). See the paragraph below for the special case of the WAIT/FWAIT instruction. 


¢ The processor executed a WAIT/FWAIT instruction while the MP and TS flags of register CRO were set, 
regardless of the setting of the EM flag. 


¢ The processor executed an x87 FPU, MMX, or SSE/SSE2/SSE3 instruction (with the exception of MOVNTI, 
PAUSE, PREFETCHh, SFENCE, LFENCE, MFENCE, and CLFLUSH) while the TS flag in control register CRO was set 
and the EM flag is clear. 


The EM flag is set when the processor does not have an internal x87 FPU floating-point unit. A device-not-available 
exception is then generated each time an x87 FPU floating-point instruction is encountered, allowing an exception 
handler to call floating-point instruction emulation routines. 


The TS flag indicates that a context switch (task switch) has occurred since the last time an x87 floating-point, 
MMX, or SSE/SSE2/SSE3 instruction was executed; but that the context of the x87 FPU, XMM, and MXCSR registers 
were not saved. When the TS flag is set and the EM flag is clear, the processor generates a device-not-available 
exception each time an x87 floating-point, MMX, or SSE/SSE2/SSE3 instruction is encountered (with the exception 
of the instructions listed above). The exception handler can then save the context of the x87 FPU, XMM, and MXCSR 
registers before it executes the instruction. See Section 2.5, “Control Registers,” for more information about the TS 
flag. 


The MP flag in control register CRO is used along with the TS flag to determine if WAIT or FWAIT instructions should 
generate a device-not-available exception. It extends the function of the TS flag to the WAIT and FWAIT instruc- 
tions, giving the exception handler an opportunity to save the context of the x87 FPU before the WAIT or FWAIT 
instruction is executed. The MP flag is provided primarily for use with the Intel 286 and Intel386 DX processors. For 
programs running on the Pentium 4, Intel Xeon, P6 family, Pentium, or Intel486 DX processors, or the Intel 487 SX 
coprocessors, the MP flag should always be set; for programs running on the Intel486 SX processor, the MP flag 
should be clear. 


Exception Error Code 
None. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the floating-point instruction or the WAIT/FWAIT instruction 
that generated the exception. 


Program State Change 


A program-state change does not accompany a device-not-available fault, because the instruction that generated 
the exception is not executed. 


If the EM flag is set, the exception handler can then read the floating-point instruction pointed to by the EIP and call 
the appropriate emulation routine. 


If the MP and TS flags are set or the TS flag alone is set, the exception handler can save the context of the x87 FPU, 
clear the TS flag, and continue execution at the interrupted floating-point or WAIT/FWAIT instruction. 
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Interrupt 8—Double Fault Exception (#DF) 


Exception Class Abort. 


Description 


Indicates that the processor detected a second exception while calling an exception handler for a prior exception. 
Normally, when the processor detects another exception while trying to call an exception handler, the two excep- 
tions can be handled serially. If, however, the processor cannot handle them serially, it signals the double-fault 
exception. To determine when two faults need to be signalled as a double fault, the processor divides the excep- 
tions into three classes: benign exceptions, contributory exceptions, and page faults (see Table 6-4). 


Table 6-4. Interrupt and Exception Classes 
Class Vector Number Description 


Benign Exceptions and Interrupts 1 Debug 

2 NMI Interrupt 

3 Breakpoint 

4 Overflow 

5 BOUND Range Exceeded 
6 Invalid Opcode 

7 Device Not Available 

9 Coprocessor Segment Overrun 
16 Floating-Point Error 

17 Alignment Check 

18 Machine Check 

19 SIMD floating-point 

All INT n 

All INTR 

0 

10 

11 

12 

13 


Divide Error 

Invalid TSS 

Segment Not Present 
Stack Fault 

General Protection 


Contributory Exceptions 


21 Control Protection 


Page Faults 14 Page Fault 
20 Virtualization Exception 


Table 6-5 shows the various combinations of exception classes that cause a double fault to be generated. A double- 
fault exception falls in the abort class of exceptions. The program or task cannot be restarted or resumed. The 
double-fault handler can be used to collect diagnostic information about the state of the machine and/or, when 
possible, to shut the application and/or system down gracefully or restart the system. 


Vol. 3A 6-33 


INTERRUPT AND EXCEPTION HANDLING 


A segment or page fault may be encountered while prefetching instructions; however, this behavior is outside the 
domain of Table 6-5. Any further faults generated while the processor is attempting to transfer control to the appro- 
priate fault handler could still lead to a double-fault sequence. 


Table 6-5. Conditions for Generating a Double Fault 


; ; Second Exception 
First Exception : : 
Benign Contributory Page Fault 
Benign Handle Exceptions Serially Handle Exceptions Serially Handle Exceptions Serially 
Contributory Handle Exceptions Serially Generate a Double Fault Handle Exceptions Serially 
Page Fault Handle Exceptions Serially Generate a Double Fault Generate a Double Fault 
Double Fault Handle Exceptions Serially Enter Shutdown Mode Enter Shutdown Mode 


If another contributory or page fault exception occurs while attempting to call the double-fault handler, the 
processor enters shutdown mode. This mode is similar to the state following execution of an HLT instruction. In this 
mode, the processor stops executing instructions until an NMI interrupt, SMI interrupt, hardware reset, or INIT# is 
received. The processor generates a special bus cycle to indicate that it has entered shutdown mode. Software 
designers may need to be aware of the response of hardware when it goes into shutdown mode. For example, hard- 
ware may turn on an indicator light on the front panel, generate an NMI interrupt to record diagnostic information, 
invoke reset initialization, generate an INIT initialization, or generate an SMI. If any events are pending during 
shutdown, they will be handled after an wake event from shutdown is processed (for example, A20M# interrupts). 


If a shutdown occurs while the processor is executing an NMI interrupt handler, then only a hardware reset can 
restart the processor. Likewise, if the shutdown occurs while executing in SMM, a hardware reset must be used to 
restart the processor. 


Exception Error Code 
Zero. The processor always pushes an error code of 0 onto the stack of the double-fault handler. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers are undefined. 


Program State Change 


A program-state following a double-fault exception is undefined. The program or task cannot be resumed or 
restarted. The only available action of the double-fault exception handler is to collect all possible context informa- 
tion for use in diagnostics and then close the application and/or shut down or reset the processor. 


If the double fault occurs when any portion of the exception handling machine state is corrupted, the handler 
cannot be invoked and the processor must be reset. 
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Interrupt 9—Coprocessor Segment Overrun 


Exception Class Abort. (Intel reserved; do not use. Recent IA-32 processors do not generate this 
exception.) 


Description 


Indicates that an Intel386 CPU-based systems with an Intel 387 math coprocessor detected a page or segment 
violation while transferring the middle portion of an Intel 387 math coprocessor operand. The P6 family, Pentium, 
and Intel486 processors do not generate this exception; instead, this condition is detected with a general protec- 
tion exception (#GP), interrupt 13. 


Exception Error Code 
None. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the instruction that generated the exception. 


Program State Change 


A program-state following a coprocessor segment-overrun exception is undefined. The program or task cannot 
be resumed or restarted. The only available action of the exception handler is to save the instruction pointer and 
reinitialize the x87 FPU using the FNINIT instruction. 
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Interrupt 10—Invalid TSS Exception (#TS) 


Exception Class Fault. 


Description 


Indicates that there was an error related to a TSS. Such an error might be detected during a task switch or during 
the execution of instructions that use information from a TSS. Table 6-6 shows the conditions that cause an invalid 
TSS exception to be generated. 


Table 6-6. Invalid TSS Conditions 


Error Code Index 


Invalid Condition 


TSS segment selector index 


The TSS segment limit is less than 67H for 32-bit TSS or less than 2CH for 16-bit TSS. 


TSS segment selector index 


During an IRET task switch, the TI flag in the TSS segment selector indicates the LDT. 


TSS segment selector index 


During an IRET task switch, the TSS segment selector exceeds descriptor table limit. 


TSS segment selector index 


During an IRET task switch, the busy flag in the TSS descriptor indicates an inactive task. 


TSS segment selector index 


During a task switch, an attempt to access data in a TSS results in a limit violation or 
canonical fault. 


TSS segment selector index 


During an IRET task switch, the backlink is a NULL selector. 


TSS segment selector index 


TSS segment selector index 


During an IRET task switch, the backlink points to a descriptor which is not a busy TSS. 
The new TSS descriptor is beyond the GDT limit. 


TSS segment selector index 


The new TSS selector is null on an attempt to lock the new TSS. 


TSS segment selector index 


TSS segment selector index 


The new TSS selector has the TI bit set on an attempt to lock the new TSS. 


The new TSS descriptor is not an available TSS descriptor on an attempt to lock the new 
TSS. 


LDT segment selector index 


LDT not valid or not present. 


Stack segment selector index 


The stack segment selector exceeds descriptor table limit. 


Stack segment selector index 


The stack segment selector is NULL. 


Stack segment selector index 


The stack segment descriptor is a non-data segment. 


Stack segment selector index 


The stack segment is not writable. 


Stack segment selector index 


The stack segment DPL +# CPL. 


Stack segment selector index 


Code segment selector index 


The stack segment selector RPL # CPL. 


The code segment selector exceeds descriptor table limit. 


Code segment selector index 


The code segment selector is NULL. 


Code segment selector index 


The code segment descriptor is not a code segment type. 


Code segment selector index 


The nonconforming code segment DPL + CPL. 


Code segment selector index 


The conforming code segment DPL is greater than CPL. 


Data segment selector index 


The data segment selector exceeds the descriptor table limit. 


Data segment selector index 


The data segment descriptor is not a readable code or data type. 


Data segment selector index 


The data segment descriptor is a nonconforming code type and RPL > DPL. 


Data segment selector index 


TSS segment selector index 


The data segment descriptor is a nonconforming code type and CPL > DPL. 


The TSS segment descriptor/upper descriptor is beyond the GDT segment limit. 


TSS segment selector index 


The TSS segment descriptor is not an available TSS type. 


TSS segment selector index 
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Table 6-6. Invalid TSS Conditions (Contd.) 


Error Code Index Invalid Condition 
TSS segment selector index The TSS segment upper descriptor is not the correct type. 
TSS segment selector index The TSS segment descriptor contains a non-canonical base. 


This exception can generated either in the context of the original task or in the context of the new task (see Section 
7.3, “Task Switching”). Until the processor has completely verified the presence of the new TSS, the exception is 
generated in the context of the original task. Once the existence of the new TSS is verified, the task switch is 
considered complete. Any invalid-TSS conditions detected after this point are handled in the context of the new 
task. (A task switch is considered complete when the task register is loaded with the segment selector for the new 
TSS and, if the switch is due to a procedure call or interrupt, the previous task link field of the new TSS references 
the old TSS.) 


The invalid-TSS handler must be a task called using a task gate. Handling this exception inside the faulting TSS 
context is not recommended because the processor state may not be consistent. 


Exception Error Code 


An error code containing the segment selector index for the segment descriptor that caused the violation is pushed 
onto the stack of the exception handler. If the EXT flag is set, it indicates that the exception was caused by an event 
external to the currently running program (for example, if an external interrupt handler using a task gate 
attempted a task switch to an invalid TSS). 


Saved Instruction Pointer 


If the exception condition was detected before the task switch was carried out, the saved contents of CS and EIP 
registers point to the instruction that invoked the task switch. If the exception condition was detected after the task 
switch was carried out, the saved contents of CS and EIP registers point to the first instruction of the new task. 


Program State Change 


The ability of the invalid-TSS handler to recover from the fault depends on the error condition than causes the fault. 
See Section 7.3, “Task Switching,” for more information on the task switch process and the possible recovery 
actions that can be taken. 


If an invalid TSS exception occurs during a task switch, it can occur before or after the commit-to-new-task point. 
If it occurs before the commit point, no program state change occurs. If it occurs after the commit point (when the 
segment descriptor information for the new segment selectors have been loaded in the segment registers), the 
processor will load all the state information from the new TSS before it generates the exception. During a task 
switch, the processor first loads all the segment registers with segment selectors from the TSS, then checks their 
contents for validity. If an invalid TSS exception is discovered, the remaining segment registers are loaded but not 
checked for validity and therefore may not be usable for referencing memory. The invalid TSS handler should not 
rely on being able to use the segment selectors found in the CS, SS, DS, ES, FS, and GS registers without causing 
another exception. The exception handler should load all segment registers before trying to resume the new task; 
otherwise, general-protection exceptions (#GP) may result later under conditions that make diagnosis more diffi- 
cult. The Intel recommended way of dealing situation is to use a task for the invalid TSS exception handler. The task 
switch back to the interrupted task from the invalid-TSS exception-handler task will then cause the processor to 
check the registers as it loads them from the TSS. 
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Interrupt 11—Segment Not Present (#NP) 


Exception Class Fault. 


Description 


Indicates that the present flag of a segment or gate descriptor is clear. The processor can generate this exception 
during any of the following operations: 


¢ While attempting to load CS, DS, ES, FS, or GS registers. [Detection of a not-present segment while loading the 
SS register causes a stack fault exception (#SS) to be generated.] This situation can occur while performing a 
task switch. 


¢ While attempting to load the LDTR using an LLDT instruction. Detection of a not-present LDT while loading the 
LDTR during a task switch operation causes an invalid-TSS exception (#TS) to be generated. 


¢ When executing the LTR instruction and the TSS is marked not present. 
¢ While attempting to use a gate descriptor or TSS that is marked segment-not-present, but is otherwise valid. 


An operating system typically uses the segment-not-present exception to implement virtual memory at the 
segment level. If the exception handler loads the segment and returns, the interrupted program or task resumes 
execution. 


A not-present indication in a gate descriptor, however, does not indicate that a segment is not present (because 
gates do not correspond to segments). The operating system may use the present flag for gate descriptors to 
trigger exceptions of special significance to the operating system. 


A contributory exception or page fault that subsequently referenced a not-present segment would cause a double 
fault (#DF) to be generated instead of #NP. 


Exception Error Code 


An error code containing the segment selector index for the segment descriptor that caused the violation is pushed 
onto the stack of the exception handler. If the EXT flag is set, it indicates that the exception resulted from either: 


* an external event (NMI or INTR) that caused an interrupt, which subsequently referenced a not-present 
segment 


® a benign exception that subsequently referenced a not-present segment 


The IDT flag is set if the error code refers to an IDT entry. This occurs when the IDT entry for an interrupt being 
serviced references a not-present gate. Such an event could be generated by an INT instruction or a hardware 
interrupt. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers normally point to the instruction that generated the exception. If the 
exception occurred while loading segment descriptors for the segment selectors in a new TSS, the CS and EIP 
registers point to the first instruction in the new task. If the exception occurred while accessing a gate descriptor, 
the CS and EIP registers point to the instruction that invoked the access (for example a CALL instruction that refer- 
ences a call gate). 


Program State Change 


If the segment-not-present exception occurs as the result of loading a register (CS, DS, SS, ES, FS, GS, or LDTR), 
a program-state change does accompany the exception because the register is not loaded. Recovery from this 
exception is possible by simply loading the missing segment into memory and setting the present flag in the 
segment descriptor. 


If the segment-not-present exception occurs while accessing a gate descriptor, a program-state change does not 
accompany the exception. Recovery from this exception is possible merely by setting the present flag in the gate 
descriptor. 


If a segment-not-present exception occurs during a task switch, it can occur before or after the commit-to-new- 
task point (see Section 7.3, “Task Switching”). If it occurs before the commit point, no program state change 
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occurs. If it occurs after the commit point, the processor will load all the state information from the new TSS 
(without performing any additional limit, present, or type checks) before it generates the exception. The segment- 
not-present exception handler should not rely on being able to use the segment selectors found in the CS, SS, DS, 
ES, FS, and GS registers without causing another exception. (See the Program State Change description for “Inter- 
rupt 10—Invalid TSS Exception (#TS)” in this chapter for additional information on how to handle this situation.) 


Vol. 3A 6-39 


INTERRUPT AND EXCEPTION HANDLING 


Interrupt 12—Stack Fault Exception (#SS) 


Exception Class Fault. 


Description 


Indicates that one of the following stack related conditions was detected: 


¢ A limit violation is detected during an operation that refers to the SS register. Operations that can cause a limit 
violation include stack-oriented instructions such as POP, PUSH, CALL, RET, IRET, ENTER, and LEAVE, as well as 
other memory references which implicitly or explicitly use the SS register (for example, MOV AX, [BP+6] or 
MOV AX, SS:[EAX+6]). The ENTER instruction generates this exception when there is not enough stack space 
for allocating local variables. 


¢ A not-present stack segment is detected when attempting to load the SS register. This violation can occur 
during the execution of a task switch, a CALL instruction to a different privilege level, a return to a different 
privilege level, an LSS instruction, or a MOV or POP instruction to the SS register. 


¢ Acanonical violation is detected in 64-bit mode during an operation that reference memory using the stack 
pointer register containing a non-canonical memory address. 


Recovery from this fault is possible by either extending the limit of the stack segment (in the case of a limit viola- 
tion) or loading the missing stack segment into memory (in the case of a not-present violation. 


In the case of a canonical violation that was caused intentionally by software, recovery is possible by loading the 
correct canonical value into RSP. Otherwise, a canonical violation of the address in RSP likely reflects some register 
corruption in the software. 


Exception Error Code 


If the exception is caused by a not-present stack segment or by overflow of the new stack during an inter-privilege- 
level call, the error code contains a segment selector for the segment that caused the exception. Here, the excep- 
tion handler can test the present flag in the segment descriptor pointed to by the segment selector to determine 
the cause of the exception. For a normal limit violation (on a stack segment already in use) the error code is set to 
0. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers generally point to the instruction that generated the exception. 
However, when the exception results from attempting to load a not-present stack segment during a task switch, the 
CS and EIP registers point to the first instruction of the new task. 


Program State Change 


A program-state change does not generally accompany a stack-fault exception, because the instruction that gener- 
ated the fault is not executed. Here, the instruction can be restarted after the exception handler has corrected the 
stack fault condition. 


If a stack fault occurs during a task switch, it occurs after the commit-to-new-task point (see Section 7.3, “Task 
Switching”). Here, the processor loads all the state information from the new TSS (without performing any addi- 
tional limit, present, or type checks) before it generates the exception. The stack fault handler should thus not rely 
on being able to use the segment selectors found in the CS, SS, DS, ES, FS, and GS registers without causing 
another exception. The exception handler should check all segment registers before trying to resume the new 
task; otherwise, general protection faults may result later under conditions that are more difficult to diagnose. (See 
the Program State Change description for “Interrupt 10—Invalid TSS Exception (#TS)” in this chapter for additional 
information on how to handle this situation.) 
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Interrupt 13—General Protection Exception (#GP) 


Exception Class Fault. 


Description 


Indicates that the processor detected one of a class of protection violations called “general-protection violations.” 
The conditions that cause this exception to be generated comprise all the protection violations that do not cause 
other exceptions to be generated (such as, invalid-TSS, segment-not-present, stack-fault, or page-fault excep- 
tions). The following conditions cause general-protection exceptions to be generated: 


Exceeding the segment limit when accessing the CS, DS, ES, FS, or GS segments. 


Exceeding the segment limit when referencing a descriptor table (except during a task switch or a stack 
switch). 


Transferring execution to a segment that is not executable. 
Writing to a code segment or a read-only data segment. 
Reading from an execute-only code segment. 


Loading the SS register with a segment selector for a read-only segment (unless the selector comes from a TSS 
during a task switch, in which case an invalid-TSS exception occurs). 


Loading the SS, DS, ES, FS, or GS register with a segment selector for a system segment. 

Loading the DS, ES, FS, or GS register with a segment selector for an execute-only code segment. 
Loading the SS register with the segment selector of an executable segment or a null segment selector. 
Loading the CS register with a segment selector for a data segment or a null segment selector. 
Accessing memory using the DS, ES, FS, or GS register when it contains a null segment selector. 
Switching to a busy task during a call or jump to a TSS. 


Using a segment selector on a non-IRET task switch that points to a TSS descriptor in the current LDT. TSS 
descriptors can only reside in the GDT. This condition causes a #TS exception during an IRET task switch. 


Violating any of the privilege rules described in Chapter 5, “Protection.” 


Exceeding the instruction length limit of 15 bytes (this only can occur when redundant prefixes are placed 
before an instruction). 


Loading the CRO register with a set PG flag (paging enabled) and a clear PE flag (protection disabled). 
Loading the CRO register with a set NW flag and a clear CD flag. 
Referencing an entry in the IDT (following an interrupt or exception) that is not an interrupt, trap, or task gate. 


Attempting to access an interrupt or exception handler through an interrupt or trap gate from virtual-8086 
mode when the handler’s code segment DPL is greater than 0. 


Attempting to write a 1 into a reserved bit of CR4. 


Attempting to execute a privileged instruction when the CPL is not equal to 0 (see Section 5.9, “Privileged 
Instructions,” for a list of privileged instructions). 


Attempting to execute SGDT, SIDT, SLDT, SMSW, or STR when CR4.UMIP = 1 and the CPL is not equal to 0. 
Writing to a reserved bit in an MSR. 
Accessing a gate that contains a null segment selector. 


Executing the INT n instruction when the CPL is greater than the DPL of the referenced interrupt, trap, or task 
gate. 


The segment selector in a call, interrupt, or trap gate does not point to a code segment. 


The segment selector operand in the LLDT instruction is a local type (TI flag is set) or does not point to a 
segment descriptor of the LDT type. 


The segment selector operand in the LTR instruction is local or points to a TSS that is not available. 
The target code-segment selector for a call, jump, or return is null. 
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¢ If the PAE and/or PSE flag in control register CR4 is set and the processor detects any reserved bits in a page- 
directory-pointer-table entry set to 1. These bits are checked during a write to control registers CRO, CR3, or 
CR4 that causes a reloading of the page-directory-pointer-table entry. 


¢ Attempting to write a non-zero value into the reserved bits of the MXCSR register. 


° Executing an SSE/SSE2/SSE3 instruction that attempts to access a 128-bit memory location that is not aligned 
on a 16-byte boundary when the instruction requires 16-byte alignment. This condition also applies to the stack 
segment. 


A program or task can be restarted following any general-protection exception. If the exception occurs while 
attempting to call an interrupt handler, the interrupted program can be restartable, but the interrupt may be lost. 


Exception Error Code 


The processor pushes an error code onto the exception handler's stack. If the fault condition was detected while 
loading a segment descriptor, the error code contains a segment selector to or IDT vector number for the 
descriptor; otherwise, the error code is 0. The source of the selector in an error code may be any of the following: 


¢ An operand of the instruction. 

¢ Aselector from a gate which is the operand of the instruction. 
¢ A selector from a TSS involved in a task switch. 

¢ IDT vector number. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the instruction that generated the exception. 


Program State Change 


In general, a program-state change does not accompany a general-protection exception, because the invalid 
instruction or operation is not executed. An exception handler can be designed to correct all of the conditions that 
cause general-protection exceptions and restart the program or task without any loss of program continuity. 


If a general-protection exception occurs during a task switch, it can occur before or after the commit-to-new-task 
point (see Section 7.3, “Task Switching”). If it occurs before the commit point, no program state change occurs. If 
it occurs after the commit point, the processor will load all the state information from the new TSS (without 
performing any additional limit, present, or type checks) before it generates the exception. The general-protection 
exception handler should thus not rely on being able to use the segment selectors found in the CS, SS, DS, ES, FS, 
and GS registers without causing another exception. (See the Program State Change description for “Interrupt 
10—Invalid TSS Exception (#TS)” in this chapter for additional information on how to handle this situation. ) 


General Protection Exception in 64-bit Mode 

The following conditions cause general-protection exceptions in 64-bit mode: 

¢ If the memory address is in a non-canonical form. 

¢ If asegment descriptor memory address is in non-canonical form. 

e If the target offset in a destination operand of a call or jmp is in a non-canonical form. 
° If acode segment or 64-bit call gate overlaps non-canonical space. 


¢ If the code segment descriptor pointed to by the selector in the 64-bit gate doesn't have the L-bit set and the 
D-bit clear. 


e If the EFLAGS.NT bit is set in IRET. 

e If the stack segment selector of IRET is null when going back to compatibility mode. 

e If the stack segment selector of IRET is null going back to CPL3 and 64-bit mode. 

¢ If a null stack segment selector RPL of IRET is not equal to CPL going back to non-CPL3 and 64-bit mode. 
° If the proposed new code segment descriptor of IRET has both the D-bit and the L-bit set. 
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If the segment descriptor pointed to by the segment selector in the destination operand is a code segment and 
it has both the D-bit and the L-bit set. 


If the segment descriptor from a 64-bit call gate is in non-canonical space. 

If the DPL from a 64-bit call-gate is less than the CPL or than the RPL of the 64-bit call-gate. 
If the type field of the upper 64 bits of a 64-bit call gate is not 0. 

If an attempt is made to load a null selector in the SS register in compatibility mode. 

If an attempt is made to load null selector in the SS register in CPL3 and 64-bit mode. 


If an attempt is made to load a null selector in the SS register in non-CPL3 and 64-bit mode where RPL is not 
equal to CPL. 


If an attempt is made to clear CRO.PG while IA-32e mode is enabled. 
If an attempt is made to set a reserved bit in CR3, CR4 or CR8. 
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Interrupt 14—Page-Fault Exception (HPF) 


Exception Class Fault. 


Description 


Indicates that, with paging enabled (the PG flag in the CRO register is set), the processor detected one of the 
following conditions while using the page-translation mechanism to translate a linear address to a physical 
address: 


The P (present) flag in a page-directory or page-table entry needed for the address translation is clear, 
indicating that a page table or the page containing the operand is not present in physical memory. 


The procedure does not have sufficient privilege to access the indicated page (that is, a procedure running in 
user mode attempts to access a supervisor-mode page). If the SMAP flag is set in CR4, a page fault may also 
be triggered by code running in supervisor mode that tries to access data at a user-mode address. If either the 
PKE flag or the PKS flag is set in CR4, the protection-key rights registers may cause page faults on data 
accesses to linear addresses with certain protection keys. 


Code running in user mode attempts to write to a read-only page. If the WP flag is set in CRO, the page fault 
will also be triggered by code running in supervisor mode that tries to write to a read-only page. 


An instruction fetch to a linear address that translates to a physical address in a memory page with the 
execute-disable bit set (for information about the execute-disable bit, see Chapter 4, “Paging”). If the SMEP 
flag is set in CR4, a page fault will also be triggered by code running in supervisor mode that tries to fetch an 
instruction from a user-mode address. 


One or more reserved bits in paging-structure entry are set to 1. See description below of RSVD error code flag. 


A shadow-stack access is made to a page that is not a shadow-stack page. See Section 18.2, “Shadow Stacks” 
in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 and Chapter 4.6, “Access 
Rights.” 


An enclave access violates one of the specified access-control requirements. See Section 33.3, “Access-control 
Requirements” and Section 33.20, “Enclave Page Cache Map (EPCM)” in Chapter 33, “Enclave Access Control 
and Data Structures.” In this case, the exception is called an SGX-induced page fault. The processor uses the 
error code (below) to distinguish SGX-induced page faults from ordinary page faults. 


The exception handler can recover from page-not-present conditions and restart the program or task without any 
loss of program continuity. It can also restart the program or task after a privilege violation, but the problem that 
caused the privilege violation may be uncorrectable. 


See also: Section 4.7, “Page-Fault Exceptions.” 


Exception Error Code 


Yes (special format). The processor provides the page-fault handler with two items of information to aid in diag- 
nosing the exception and recovering from it: 


An error code on the stack. The error code for a page fault has a format different from that for other exceptions 

(see Figure 6-11). The processor establishes the bits in the error code as follows: 

— P flag (bit 0). 
This flag is O if there is no translation for the linear address because the P flag was 0 in one of the paging- 
structure entries used to translate that address. 

— WYR (bit 1). 
If the access causing the page-fault exception was a write, this flag is 1; otherwise, it is 0. This flag 
describes the access causing the page-fault exception, not the access rights specified by paging. 

— U/S (bit 2). 
If a user-mode access caused the page-fault exception, this flag is 1; it is 0 if a supervisor-mode access did 
so. This flag describes the access causing the page-fault exception, not the access rights specified by 
paging. 
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RSVD flag (bit 3). 
This flag is 1 if there is no translation for the linear address because a reserved bit was set in one of the 
paging-structure entries used to translate that address. 


I/D flag (bit 4). 
This flag is 1 if the access causing the page-fault exception was an instruction fetch. This flag describes the 
access causing the page-fault exception, not the access rights specified by paging. 


PK flag (bit 5). 
This flag is 1 if the access causing the page-fault exception was a data access to a linear address with a 
protection key for which the protection-key rights registers disallow access. 


SS (bit 1). 

If the access causing the page-fault exception was a shadow-stack access (including shadow-stack 
accesses in enclave mode), this flag is 1; otherwise, it is 0. This flag describes the access causing the page- 
fault exception, not the access rights specified by paging. 


SGX flag (bit 15). 

This flag is 1 if the exception is unrelated to paging and resulted from violation of SGX-specific access- 
control requirements. Because such a violation can occur only if there is no ordinary page fault, this flag is 
set only if the P flag (bit 0) is 1 and the RSVD flag (bit 3) and the PK flag (bit 5) are both 0. 


See Section 4.6, “Access Rights” and Section 4.7, “Page-Fault Exceptions” for more information about page- 
fault exceptions and the error codes that they produce. 


31 15 6543210 
8 alzsaigs” 
Reserved >< Reserved ¥ =|" |7 
P 0 The fault was caused by a non-present page. 


1 The fault was caused by a page-level protection violation. 


W/R 0 The access causing the fault was a read. 
1 The access causing the fault was a write. 


U/S 0 A supervisor-mode access caused the fault. 
1 Auser-mode access caused the fault. 


RSVD 0 The fault was not caused by reserved bit violation. 
1 The fault was caused by a reserved bit set to 1 in some 
paging-structure entry. 


/D 0 The fault was not caused by an instruction fetch. 
1 The fault was caused by an instruction fetch. 


PK 0 The fault was not caused by protection keys. 
1 There was a protection-key violation. 


ss 0 The fault was not caused by a shadow-stack access. 
1 The fault was caused by a shadow-stack access. 


SGX 0 The fault is not related to SGX. 
1 The fault resulted from violation of SGX-specific access-control 
requirements. 


Figure 6-11. Page-Fault Error Code 


The contents of the CR2 register. The processor loads the CR2 register with the 32-bit linear address that 
generated the exception. The page-fault handler can use this address to locate the corresponding page 
directory and page-table entries. Another page fault can potentially occur during execution of the page-fault 
handler; the handler should save the contents of the CR2 register before a second page fault can occur.® If a 
page fault is caused by a page-level protection violation, the access flag in the page-directory entry is set when 
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the fault occurs. The behavior of IA-32 processors regarding the access flag in the corresponding page-table 
entry is model specific and not architecturally defined. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers generally point to the instruction that generated the exception. If the 
page-fault exception occurred during a task switch, the CS and EIP registers may point to the first instruction of the 
new task (as described in the following “Program State Change” section). 


Program State Change 


A program-state change does not normally accompany a page-fault exception, because the instruction that causes 
the exception to be generated is not executed. After the page-fault exception handler has corrected the violation 
(for example, loaded the missing page into memory), execution of the program or task can be resumed. 


When a page-fault exception is generated during a task switch, the program-state may change, as follows. During 
a task switch, a page-fault exception can occur during any of following operations: 


¢ While writing the state of the original task into the TSS of that task. 

¢ While reading the GDT to locate the TSS descriptor of the new task. 

¢ While reading the TSS of the new task. 

¢ While reading segment descriptors associated with segment selectors from the new task. 

¢ While reading the LDT of the new task to verify the segment registers stored in the new TSS. 


In the last two cases the exception occurs in the context of the new task. The instruction pointer refers to the first 
instruction of the new task, not to the instruction which caused the task switch (or the last instruction to be 
executed, in the case of an interrupt). If the design of the operating system permits page faults to occur during 
task-switches, the page-fault handler should be called through a task gate. 


If a page fault occurs during a task switch, the processor will load all the state information from the new TSS 
(without performing any additional limit, present, or type checks) before it generates the exception. The page-fault 
handler should thus not rely on being able to use the segment selectors found in the CS, SS, DS, ES, FS, and GS 
registers without causing another exception. (See the Program State Change description for “Interrupt 10—Invalid 
TSS Exception (#TS)” in this chapter for additional information on how to handle this situation.) 


Additional Exception-Handling Information 


Special care should be taken to ensure that an exception that occurs during an explicit stack switch does not cause 
the processor to use an invalid stack pointer (SS:ESP). Software written for 16-bit IA-32 processors often use a 
pair of instructions to change to a new stack, for example: 


MOV SS, AX 
MOV SP, StackTop 


When executing this code on one of the 32-bit IA-32 processors, it is possible to get a page fault, general-protec- 
tion fault (#GP), or alignment check fault (#AC) after the segment selector has been loaded into the SS register 
but before the ESP register has been loaded. At this point, the two parts of the stack pointer (SS and ESP) are 
inconsistent. The new stack segment is being used with the old stack pointer. 


The processor does not use the inconsistent stack pointer if the exception handler switches to a well defined stack 
(that is, the handler is a task or a more privileged procedure). However, if the exception handler is called at the 
same privilege level and from the same task, the processor will attempt to use the inconsistent stack pointer. 


In systems that handle page-fault, general-protection, or alignment check exceptions within the faulting task (with 
trap or interrupt gates), software executing at the same privilege level as the exception handler should initialize a 
new stack by using the LSS instruction rather than a pair of MOV instructions, as described earlier in this note. 
When the exception handler is running at privilege level 0 (the normal case), the problem is limited to procedures 
or tasks that run at privilege level 0, typically the kernel of the operating system. 


6. Processors update CR2 whenever a page fault is detected. If a second page fault occurs while an earlier page fault is being deliv- 
ered, the faulting linear address of the second fault will overwrite the contents of CR2 (replacing the previous address). These 
updates to CR2 occur even if the page fault results in a double fault or occurs during the delivery of a double fault. 
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Interrupt 16—x87 FPU Floating-Point Error (HMF) 


Exception Class Fault. 


Description 


Indicates that the x87 FPU has detected a floating-point error. The NE flag in the register CRO must be set for an 
interrupt 16 (floating-point error exception) to be generated. (See Section 2.5, “Control Registers,” for a detailed 
description of the NE flag.) 


NOTE 
SIMD floating-point exceptions (#XM) are signaled through interrupt 19. 


While executing x87 FPU instructions, the x87 FPU detects and reports six types of floating-point error conditions: 
¢ Invalid operation (#1) 
— Stack overflow or underflow (#IS) 
— Invalid arithmetic operation (#IA) 
* Divide-by-zero (#Z) 
¢ Denormalized operand (#D) 
¢ Numeric overflow (#0) 
¢ Numeric underflow (#U) 
¢  Inexact result (precision) (#P) 


Each of these error conditions represents an x87 FPU exception type, and for each of exception type, the x87 FPU 
provides a flag in the x87 FPU status register and a mask bit in the x87 FPU control register. If the x87 FPU detects 
a floating-point error and the mask bit for the exception type is set, the x87 FPU handles the exception automati- 
cally by generating a predefined (default) response and continuing program execution. The default responses have 
been designed to provide a reasonable result for most floating-point applications. 


If the mask for the exception is clear and the NE flag in register CRO is set, the x87 FPU does the following: 
1. Sets the necessary flag in the FPU status register. 


2. Waits until the next “waiting” x87 FPU instruction or WAIT/FWAIT instruction is encountered in the program’s 
instruction stream. 


3. Generates an internal error signal that cause the processor to generate a floating-point exception (#MF). 


Prior to executing a waiting x87 FPU instruction or the WAIT/FWAIT instruction, the x87 FPU checks for pending 
x87 FPU floating-point exceptions (as described in step 2 above). Pending x87 FPU floating-point exceptions are 
ignored for “non-waiting” x87 FPU instructions, which include the FNINIT, FNCLEX, FNSTSW, FNSTSW AX, FNSTCW, 
FNSTENV, and FNSAVE instructions. Pending x87 FPU exceptions are also ignored when executing the state 
management instructions FXSAVE and FXRSTOR. 


All of the x87 FPU floating-point error conditions can be recovered from. The x87 FPU floating-point-error exception 
handler can determine the error condition that caused the exception from the settings of the flags in the x87 FPU 
status word. See “Software Exception Handling” in Chapter 8 of the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1, for more information on handling x87 FPU floating-point exceptions. 


Exception Error Code 
None. The x87 FPU provides its own error information. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the floating-point or WAIT/FWAIT instruction that was about to 
be executed when the floating-point-error exception was generated. This is not the faulting instruction in which the 
error condition was detected. The address of the faulting instruction is contained in the x87 FPU instruction pointer 
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register. See Section 8.1.8, “x87 FPU Instruction and Data (Operand) Pointers” in Chapter 8 of the Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 1, for more information about information the FPU saves 
for use in handling floating-point-error exceptions. 


Program State Change 


A program-state change generally accompanies an x87 FPU floating-point exception because the handling of the 
exception is delayed until the next waiting x87 FPU floating-point or WAIT/FWAIT instruction following the faulting 
instruction. The x87 FPU, however, saves sufficient information about the error condition to allow recovery from the 
error and re-execution of the faulting instruction if needed. 


In situations where non- x87 FPU floating-point instructions depend on the results of an x87 FPU floating-point 
instruction, a WAIT or FWAIT instruction can be inserted in front of a dependent instruction to force a pending x87 
FPU floating-point exception to be handled before the dependent instruction is executed. See “x87 FPU Exception 
Synchronization” in Chapter 8 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, 
for more information about synchronization of x87 floating-point-error exceptions. 
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Interrupt 17—Alignment Check Exception (#AC) 


Exception Class Fault. 


Description 


Indicates that the processor detected an unaligned memory operand when alignment checking was enabled. Align- 
ment checks are only carried out in data (or stack) accesses (not in code fetches or system segment accesses). An 
example of an alignment-check violation is a word stored at an odd byte address, or a doubleword stored at an 
address that is not an integer multiple of 4. Table 6-7 lists the alignment requirements various data types recog- 
nized by the processor. 


Table 6-7. Alignment Requirements by Data Type 
Data Type Address Must Be Divisible By 
Word 


Doubleword 


Single-precision floating-point (32-bits) 
Double-precision floating-point (64-bits) 
Double extended-precision floating-point (80-bits) 


Quadword 


(op) 


Double quadword 
Segment Selector 
32-bit Far Pointer 
48-bit Far Pointer 
32-bit Pointer 
GDTR, IDTR, LDTR, or Task Register Contents 4 

FSTENV/FLDENV Save Area 4 or 2, depending on operand size 
FSAVE/FRSTOR Save Area 4 or 2, depending on operand size 

Bit String 2 or 4 depending on the operand-size attribute. 


BF BNN SF DO DOO fF FN 


Note that the alignment check exception (#AC) is generated only for data types that must be aligned on word, 
doubleword, and quadword boundaries. A general-protection exception (#GP) is generated 128-bit data types that 
are not aligned on a 16-byte boundary. 


To enable alignment checking, the following conditions must be true: 

¢ AM flag in CRO register is set. 

¢ AC flag in the EFLAGS register is set. 

° The CPL is 3 (including virtual-8086 mode). 

Alignment-check exceptions (#AC) are generated only when operating at privilege level 3 (user mode). Memory 


references that default to privilege level 0, such as segment descriptor loads, do not generate alignment-check 
exceptions, even when caused by a memory reference made from privilege level 3. 


Storing the contents of the GDTR, IDTR, LDTR, or task register in memory while at privilege level 3 can generate 
an alignment-check exception. Although application programs do not normally store these registers, the fault can 
be avoided by aligning the information stored on an even word-address. 


The FXSAVE/XSAVE and FXRSTOR/XRSTOR instructions save and restore a 512-byte data structure, the first byte 
of which must be aligned on a 16-byte boundary. If the alignment-check exception (#AC) is enabled when 
executing these instructions (and CPL is 3), a misaligned memory operand can cause either an alignment-check 
exception or a general-protection exception (#GP) depending on the processor implementation (see “FXSAVE- 
Save x87 FPU, MMX, SSE, and SSE2 State” and “FXRSTOR-Restore x87 FPU, MMX, SSE, and SSE2 State” in 
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Chapter 3 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A; see “XSAVE—Save 
Processor Extended States” and “XRSTOR—Restore Processor Extended States” in Chapter 5 of the Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 2C). 


The MOVDQU, MOVUPS, and MOVUPD instructions perform 128-bit unaligned loads or stores. The LDDQU instruc- 
tions loads 128-bit unaligned data. They do not generate general-protection exceptions (#GP) when operands are 
not aligned on a 16-byte boundary. If alignment checking is enabled, alignment-check exceptions (#AC) may or 
may not be generated depending on processor implementation when data addresses are not aligned on an 8-byte 
boundary. 


FSAVE and FRSTOR instructions can generate unaligned references, which can cause alignment-check faults. These 
instructions are rarely needed by application programs. 
Exception Error Code 


Yes. The error code is null; all bits are clear except possibly bit O — EXT; see Section 6.13. EXT is set if the #AC is 
recognized during delivery of an event other than a software interrupt (see “INT n/INTO/INT3/INT1—Call to Inter- 
rupt Procedure” in Chapter 3 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A). 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the instruction that generated the exception. 


Program State Change 


A program-state change does not accompany an alignment-check fault, because the instruction is not executed. 
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Interrupt 18—Machine-Check Exception (#MC) 


Exception Class Abort. 


Description 


Indicates that the processor detected an internal machine error or a bus error, or that an external agent detected 
a bus error. The machine-check exception is model-specific, available on the Pentium and later generations of 
processors. The implementation of the machine-check exception is different between different processor families, 
and these implementations may not be compatible with future Intel 64 or IA-32 processors. (Use the CPUID 
instruction to determine whether this feature is present.) 


Bus errors detected by external agents are signaled to the processor on dedicated pins: the BINIT# and MCERR# 
pins on the Pentium 4, Intel Xeon, and P6 family processors and the BUSCHK# pin on the Pentium processor. When 
one of these pins is enabled, asserting the pin causes error information to be loaded into machine-check registers 
and a machine-check exception is generated. 


The machine-check exception and machine-check architecture are discussed in detail in Chapter 15, “Machine- 
Check Architecture.” Also, see the data books for the individual processors for processor-specific hardware infor- 
mation. 


Exception Error Code 


None. Error information is provided by machine-check MSRs. 


Saved Instruction Pointer 


For the Pentium 4 and Intel Xeon processors, the saved contents of extended machine-check state registers are 
directly associated with the error that caused the machine-check exception to be generated (see Section 15.3.1.2, 
“IA32_MCG_STATUS MSR,” and Section 15.3.2.6, “IA32_MCG Extended Machine Check State MSRs”). 


For the P6 family processors, if the EIPV flag in the MCG_STATUS MSR is set, the saved contents of CS and EIP 
registers are directly associated with the error that caused the machine-check exception to be generated; if the 
flag is clear, the saved instruction pointer may not be associated with the error (see Section 15.3.1.2, 
“IA32_MCG_STATUS MSR"), 


For the Pentium processor, contents of the CS and EIP registers may not be associated with the error. 


Program State Change 
The machine-check mechanism is enabled by setting the MCE flag in control register CR4. 


For the Pentium 4, Intel Xeon, P6 family, and Pentium processors, a program-state change always accompanies a 
machine-check exception, and an abort class exception is generated. For abort exceptions, information about the 
exception can be collected from the machine-check MSRs, but the program cannot generally be restarted. 


If the machine-check mechanism is not enabled (the MCE flag in control register CR4 is clear), a machine-check 
exception causes the processor to enter the shutdown state. 
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Interrupt 19—SIMD Floating-Point Exception (#XM) 


Exception Class Fault. 


Description 


Indicates the processor has detected an SSE/SSE2/SSE3 SIMD floating-point exception. The appropriate status 
flag in the MXCSR register must be set and the particular exception unmasked for this interrupt to be generated. 


There are six classes of numeric exception conditions that can occur while executing an SSE/ SSE2/SSE3 SIMD 
floating-point instruction: 


¢ Invalid operation (#1) 

° Divide-by-zero (#Z) 

¢ Denormal operand (#D) 

¢ Numeric overflow (#0) 

¢ Numeric underflow (#U) 

¢ Inexact result (Precision) (#P) 


The invalid operation, divide-by-zero, and denormal-operand exceptions are pre-computation exceptions; that is, 
they are detected before any arithmetic operation occurs. The numeric underflow, numeric overflow, and inexact 
result exceptions are post-computational exceptions. 


See “SIMD Floating-Point Exceptions” in Chapter 11 of the Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 1, for additional information about the SIMD floating-point exception classes. 


When a SIMD floating-point exception occurs, the processor does either of the following things: 


¢ It handles the exception automatically by producing the most reasonable result and allowing program 
execution to continue undisturbed. This is the response to masked exceptions. 


¢ It generates a SIMD floating-point exception, which in turn invokes a software exception handler. This is the 
response to unmasked exceptions. 


Each of the six SIMD floating-point exception conditions has a corresponding flag bit and mask bit in the MXCSR 
register. If an exception is masked (the corresponding mask bit in the MXCSR register is set), the processor takes 
an appropriate automatic default action and continues with the computation. If the exception is unmasked (the 
corresponding mask bit is clear) and the operating system supports SIMD floating-point exceptions (the OSXM- 
MEXCPT flag in control register CR4 is set), a software exception handler is invoked through a SIMD floating-point 
exception. If the exception is unmasked and the OSXMMEXCPT bit is clear (indicating that the operating system 
does not support unmasked SIMD floating-point exceptions), an invalid opcode exception (#UD) is signaled instead 
of a SIMD floating-point exception. 


Note that because SIMD floating-point exceptions are precise and occur immediately, the situation does not arise 
where an x87 FPU instruction, a WAIT/FWAIT instruction, or another SSE/SSE2/SSE3 instruction will catch a 
pending unmasked SIMD floating-point exception. 


In situations where a SIMD floating-point exception occurred while the SIMD floating-point exceptions were 
masked (causing the corresponding exception flag to be set) and the SIMD floating-point exception was subse- 
quently unmasked, then no exception is generated when the exception is unmasked. 


When SSE/SSE2/SSE3 SIMD floating-point instructions operate on packed operands (made up of two or four sub- 
operands), multiple SIMD floating-point exception conditions may be detected. If no more than one exception 
condition is detected for one or more sets of sub-operands, the exception flags are set for each exception condition 
detected. For example, an invalid exception detected for one sub-operand will not prevent the reporting of a divide- 
by-zero exception for another sub-operand. However, when two or more exceptions conditions are generated for 
one sub-operand, only one exception condition is reported, according to the precedences shown in Table 6-8. This 
exception precedence sometimes results in the higher priority exception condition being reported and the lower 
priority exception conditions being ignored. 
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Table 6-8. SIMD Floating-Point Exceptions Priority 


Priority Description 


1 (Highest) Invalid operation exception due to SNaN operand (or any NaN operand for maximum, minimum, or certain compare and 
convert operations). 


2 QNaN operand!. 
3 Any other invalid operation exception not mentioned above or a divide-by-zero exception®. 
4 Denormal operand exception®. 


5 Numeric overflow and underflow exceptions possibly in conjunction with the inexact result exception®. 


6 (Lowest) Inexact result exception. 


NOTES: 

1. Though a QNaN this is not an exception, the handling of a QNaN operand has precedence over lower priority exceptions. For exam- 
ple, a QNaN divided by zero results in a QNaN, not a divide-by-zero- exception. 

2. If masked, then instruction execution continues, and a lower priority exception can occur as well. 


Exception Error Code 


None. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the SSE/SSE2/SSE3 instruction that was executed when the 
SIMD floating-point exception was generated. This is the faulting instruction in which the error condition was 
detected. 


Program State Change 


A program-state change does not accompany a SIMD floating-point exception because the handling of the excep- 
tion is immediate unless the particular exception is masked. The available state information is often sufficient to 
allow recovery from the error and re-execution of the faulting instruction if needed. 
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Interrupt 20—Virtualization Exception (#VE) 


Exception Class Fault. 


Description 


Indicates that the processor detected an EPT violation in VMX non-root operation. Not all EPT violations cause virtu- 
alization exceptions. See Section 24.5.7.2 for details. 


The exception handler can recover from EPT violations and restart the program or task without any loss of program 
continuity. In some cases, however, the problem that caused the EPT violation may be uncorrectable. 


Exception Error Code 


None. 


Saved Instruction Pointer 
The saved contents of CS and EIP registers generally point to the instruction that generated the exception. 


Program State Change 


A program-state change does not normally accompany a virtualization exception, because the instruction that 
causes the exception to be generated is not executed. After the virtualization exception handler has corrected the 
violation (for example, by executing the EPTP-switching VM function), execution of the program or task can be 
resumed. 


Additional Exception-Handling Information 


The processor saves information about virtualization exceptions in the virtualization-exception information area. 
See Section 24.5.7.2 for details. 
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Interrupt 21—Control Protection Exception (#CP) 


Exception Class Fault. 


Description 
Indicates a control flow transfer attempt violated the control flow enforcement technology constraints. 


Exception Error Code 


Yes (special format). The processor provides the control protection exception handler with following information 
through the error code on the stack. 


31 15 14 0 


ENCL 


CPEC 


| 
Reserved 


Figure 6-12. Exception Error Code Information 


¢  =Bit 14:0 - CPEC 
— 1- NEAR-RET: Indicates the #CP was caused by a near RET instruction. 
— 2- FAR-RET/IRET: Indicates the #CP was caused by a FAR RET or IRET instruction. 


— 3-ENDBRANCH: indicates the #CP was due to missing ENDBRANCH at target of an indirect call or jump 
instruction. 


— 4-RSTORSSP: Indicates the #CP was caused by a shadow-stack-restore token check failure in the 
RSTORSSP instruction. 


— 5-SETSSBSY: Indicates #CP was caused by a supervisor shadow stack token check failure in the SETSSBSY 
instruction. 


¢ Bit 15 (ENCL) of the error code, if set to 1, indicates the #CP occurred during enclave execution. 


Saved Instruction Pointer 
The saved contents of the CS and EIP registers generally point to the instruction that generated the exception. 


Program State Change 


A program-state change does not normally accompany a control protection exception, because the instruction that 
causes the exception to be generated is not executed. 


When a control protection exception is generated during a task switch, the program-state may change as follows. 
During a task switch, a control protection exception can occur during any of following operations: 


¢ If task switch is initiated by IRET, CS and LIP stored on old task shadow stack do not match CS and LIP of new 
task (where LIP is the linear address of the return address). 


¢ If task switch is initiated by IRET and SSP of new task loaded from shadow stack of old task (if new task CPL is 
< 3), OR the SSP from IA32_PL3_SSP (if new task CPL = 3) is not aligned to 4 bytes or is a value beyond 4GB. 
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In these cases the exception occurs in the context of the new task. The instruction pointer refers to the first instruc- 
tion of the new task, not to the instruction which caused the task switch (or the last instruction to be executed, in 
the case of an interrupt). If the design of the operating system permits control protection faults to occur during 
task-switches, the control protection fault handler should be called through a task gate. 
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Interrupts 32 to 255—User Defined Interrupts 


Exception Class Not applicable. 


Description 
Indicates that the processor did one of the following things: 
¢ Executed an INT rn instruction where the instruction operand is one of the vector numbers from 32 through 255. 


¢ Responded to an interrupt request at the INTR pin or from the local APIC when the interrupt vector number 
associated with the request is from 32 through 255. 


Exception Error Code 
Not applicable. 


Saved Instruction Pointer 


The saved contents of CS and EIP registers point to the instruction that follows the INT rn instruction or instruction 
following the instruction on which the INTR signal occurred. 


Program State Change 


A program-state change does not accompany interrupts generated by the INT rn instruction or the INTR signal. The 
INT rn instruction generates the interrupt within the instruction stream. When the processor receives an INTR 
signal, it commits all state changes for all previous instructions before it responds to the interrupt; so, program 
execution can resume upon returning from the interrupt handler. 
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CHAPTER 7 
TASK MANAGEMENT 


This chapter describes the IA-32 architecture's task management facilities. These facilities are only available when 
the processor is running in protected mode. 


This chapter focuses on 32-bit tasks and the 32-bit TSS structure. For information on 16-bit tasks and the 16-bit 
TSS structure, see Section 7.6, “16-Bit Task-State Segment (TSS).” For information specific to task management in 
64-bit mode, see Section 7.7, “Task Management in 64-bit Mode.” 


7.1 TASK MANAGEMENT OVERVIEW 


A task is a unit of work that a processor can dispatch, execute, and suspend. It can be used to execute a program, 
a task or process, an operating-system service utility, an interrupt or exception handler, or a kernel or executive 
utility. 


The IA-32 architecture provides a mechanism for saving the state of a task, for dispatching tasks for execution, and 
for switching from one task to another. When operating in protected mode, all processor execution takes place from 
within a task. Even simple systems must define at least one task. More complex systems can use the processor’s 
task management facilities to support multitasking applications. 


7.1.1 Task Structure 


A task is made up of two parts: a task execution space and a task-state segment (TSS). The task execution space 
consists of a code segment, a stack segment, and one or more data segments (see Figure 7-1). If an operating 
system or executive uses the processor's privilege-level protection mechanism, the task execution space also 
provides a separate stack for each privilege level. 


The TSS specifies the segments that make up the task execution space and provides a storage place for task state 
information. In multitasking systems, the TSS also provides a mechanism for linking tasks. 


A task is identified by the segment selector for its TSS. When a task is loaded into the processor for execution, the 
segment selector, base address, limit, and segment descriptor attributes for the TSS are loaded into the task 
register (see Section 2.4.4, “Task Register (TR)”). 


If paging is implemented for the task, the base address of the page directory used by the task is loaded into control 
register CR3. 
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Figure 7-1. Structure of a Task 


7.1.2 Task State 


The following items define the state of the currently executing task: 


¢ The task’s current execution space, defined by the segment selectors in the segment registers (CS, DS, SS, ES, 
FS, and GS). 


¢ The state of the general-purpose registers. 

¢ The state of the EFLAGS register. 

¢ The state of the EIP register. 

¢ The state of control register CR3. 

° The state of the task register. 

¢ The state of the LDTR register. 

¢ The I/O map base address and I/O map (contained in the TSS). 

¢ Stack pointers to the privilege 0, 1, and 2 stacks (contained in the TSS). 
¢ Link to previously executed task (contained in the TSS). 

° The state of the shadow stack pointer (SSP). 


Prior to dispatching a task, all of these items are contained in the task’s TSS, except the state of the task register. 
Also, the complete contents of the LDTR register are not contained in the TSS, only the segment selector for the 
LDT. 


713 Executing a Task 


Software or the processor can dispatch a task for execution in one of the following ways: 

¢ A explicit call to a task with the CALL instruction. 

¢ Aexplicit jump to a task with the JMP instruction. 

¢ An implicit call (by the processor) to an interrupt-handler task. 

¢ An implicit call to an exception-handler task. 

¢ Areturn (initiated with an IRET instruction) when the NT flag in the EFLAGS register is set. 


All of these methods for dispatching a task identify the task to be dispatched with a segment selector that points to 
a task gate or the TSS for the task. When dispatching a task with a CALL or JMP instruction, the selector in the 
instruction may select the TSS directly or a task gate that holds the selector for the TSS. When dispatching a task 
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to handle an interrupt or exception, the IDT entry for the interrupt or exception must contain a task gate that holds 
the selector for the interrupt- or exception-handler TSS. 


When a task is dispatched for execution, a task switch occurs between the currently running task and the 
dispatched task. During a task switch, the execution environment of the currently executing task (called the task’s 
state or context) is saved in its TSS and execution of the task is suspended. The context for the dispatched task is 
then loaded into the processor and execution of that task begins with the instruction pointed to by the newly loaded 
EIP register. If the task has not been run since the system was last initialized, the EIP will point to the first instruc- 
tion of the task’s code; otherwise, it will point to the next instruction after the last instruction that the task 
executed when it was last active. 


If the currently executing task (the calling task) called the task being dispatched (the called task), the TSS 
segment selector for the calling task is stored in the TSS of the called task to provide a link back to the calling task. 


For all IA-32 processors, tasks are not recursive. A task cannot call or jump to itself. 


Interrupts and exceptions can be handled with a task switch to a handler task. Here, the processor performs a task 
switch to handle the interrupt or exception and automatically switches back to the interrupted task upon returning 
from the interrupt-handler task or exception-handler task. This mechanism can also handle interrupts that occur 
during interrupt tasks. 


As part of a task switch, the processor can also switch to another LDT, allowing each task to have a different logical- 
to-physical address mapping for LDT-based segments. The page-directory base register (CR3) also is reloaded ona 
task switch, allowing each task to have its own set of page tables. These protection facilities help isolate tasks and 
prevent them from interfering with one another. 


If protection mechanisms are not used, the processor provides no protection between tasks. This is true even with 
operating systems that use multiple privilege levels for protection. A task running at privilege level 3 that uses the 
same LDT and page tables as other privilege-level-3 tasks can access code and corrupt data and the stack of other 
tasks. 


Use of task management facilities for handling multitasking applications is optional. Multitasking can be handled in 
software, with each software defined task executed in the context of a single IA-32 architecture task. 


If shadow stack is enabled, then the SSP of the task is located at the 4 bytes at offset 104 in the 32-bit TSS and is 
used by the processor to establish the SSP when a task switch occurs from a task associated with this TSS. Note 
that the processor does not write the SSP of the task initiating the task switch to the TSS of that task, and instead 
the SSP of the previous task is pushed onto the shadow stack of the new task. 


7.2 TASK MANAGEMENT DATA STRUCTURES 


The processor defines five data structures for handling task-related activities: 
° Task-state segment (TSS). 

¢ Task-gate descriptor. 

¢ TSS descriptor. 

° Task register. 

¢ NT flag in the EFLAGS register. 


When operating in protected mode, a TSS and TSS descriptor must be created for at least one task, and the 
segment selector for the TSS must be loaded into the task register (using the LTR instruction). 


7.2.1 Task-State Segment (TSS) 


The processor state information needed to restore a task is saved in a system segment called the task-state 
segment (TSS). Figure 7-2 shows the format of a TSS for tasks designed for 32-bit CPUs. The fields of a TSS are 
divided into two main categories: dynamic fields and static fields. 


For information about 16-bit Intel 286 processor task structures, see Section 7.6, “16-Bit Task-State Segment 
(TSS).” For information about 64-bit mode task structures, see Section 7.7, “Task Management in 64-bit Mode.” 
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Figure 7-2. 32-Bit Task-State Segment (TSS) 


The processor updates dynamic fields when a task is suspended during a task switch. The following are dynamic 


fields: 


¢ General-purpose register fields — State of the EAX, ECX, EDX, EBX, ESP, EBP, ESI, and EDI registers prior 


to the task switch. 


¢ Segment selector fields — Segment selectors stored in the ES, CS, SS, DS, FS, and GS registers prior to the 


task switch. 


¢ EFLAGS register field — State of the EFLAGS register prior to the task switch. 
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¢  EIP (instruction pointer) field — State of the EIP register prior to the task switch. 


° Previous task link field — Contains the segment selector for the TSS of the previous task (updated on a task 
switch that was initiated by a call, interrupt, or exception). This field (which is sometimes called the back link 
field) permits a task switch back to the previous task by using the IRET instruction. 


The processor reads the static fields, but does not normally change them. These fields are set up when a task is 
created. The following are static fields: 


¢ LDT segment selector field — Contains the segment selector for the task's LDT. 


¢ CR3 control register field — Contains the base physical address of the page directory to be used by the task. 
Control register CR3 is also known as the page-directory base register (PDBR). 


° Privilege level-O, -1, and -2 stack pointer fields — These stack pointers consist of a logical address made 
up of the segment selector for the stack segment (SSO, SS1, and SS2) and an offset into the stack (ESPO, 
ESP1i, and ESP2). Note that the values in these fields are static for a particular task; whereas, the SS and ESP 
values will change if stack switching occurs within the task. 


° T (debug trap) flag (byte 100, bit 0) — When set, the T flag causes the processor to raise a debug exception 
when a task switch to this task occurs (see Section 17.3.1.5, “Task-Switch Exception Condition”). 


¢ I/O map base address field — Contains a 16-bit offset from the base of the TSS to the I/O permission bit 
map and interrupt redirection bitmap. When present, these maps are stored in the TSS at higher addresses. 
The I/O map base address points to the beginning of the I/O permission bit map and the end of the interrupt 
redirection bit map. See Chapter 19, “Input/Output,” in the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1, for more information about the I/O permission bit map. See Section 19.3, 
“Interrupt and Exception Handling in Virtual-8086 Mode,” for a detailed description of the interrupt redirection 
bit map. 

¢ Shadow Stack Pointer (SSP) — Contains task's shadow stack pointer. The shadow stack of the task should 
have a supervisor shadow stack token at the address pointed to by the task SSP (offset 104). This token will be 
verified and made busy when switching to that shadow stack using a CALL/JMP instruction, and made free 
when switching out of that task using an IRET instruction. 


If paging is used: 


¢ Pages corresponding to the previous task’s TSS, the current task’s TSS, and the descriptor table entries for 
each all should be marked as read/write. 


° Task switches are carried out faster if the pages containing these structures are present in memory before the 
task switch is initiated. 


7.2.2 TSS Descriptor 


The TSS, like all other segments, is defined by a segment descriptor. Figure 7-3 shows the format of a TSS 
descriptor. TSS descriptors may only be placed in the GDT; they cannot be placed in an LDT or the IDT. 


An attempt to access a TSS using a segment selector with its TI flag set (which indicates the current LDT) causes 
a general-protection exception (#GP) to be generated during CALLs and JMPs; it causes an invalid TSS exception 
(#TS) during IRETs. A general-protection exception is also generated if an attempt is made to load a segment 
selector for a TSS into a segment register. 


The busy flag (B) in the type field indicates whether the task is busy. A busy task is currently running or suspended. 
A type field with a value of 1001B indicates an inactive task; a value of 1011B indicates a busy task. Tasks are not 
recursive. The processor uses the busy flag to detect an attempt to call a task whose execution has been inter- 
rupted. To ensure that there is only one busy flag is associated with a task, each TSS should have only one TSS 
descriptor that points to it. 
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TSS Descriptor 


31 24 23 22 212019 1615141312 11 8 7 0 
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L ; L jo} | 0 | B| 1 
31 16 15 0 
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AVL Available for use by system software 
B Busy flag 

BASE Segment Base Address 

DPL Descriptor Privilege Level 


G Granularity 
LIMIT Segment Limit 
P Segment Present 


TYPE Segment Type 


Figure 7-3. TSS Descriptor 


The base, limit, and DPL fields and the granularity and present flags have functions similar to their use in data- 
segment descriptors (see Section 3.4.5, “Segment Descriptors”). When the G flag is 0 in a TSS descriptor for a 32- 
bit TSS, the limit field must have a value equal to or greater than 67H, one byte less than the minimum size of a 
TSS. Attempting to switch to a task whose TSS descriptor has a limit less than 67H generates an invalid-TSS excep- 
tion (#TS). A larger limit is required if an I/O permission bit map is included or if the operating system stores addi- 
tional data. The processor does not check for a limit greater than 67H on a task switch; however, it does check 
when accessing the I/O permission bit map or interrupt redirection bit map. 


Any program or procedure with access to a TSS descriptor (that is, whose CPL is numerically equal to or less than 
the DPL of the TSS descriptor) can dispatch the task with a call or a jump. 


In most systems, the DPLs of TSS descriptors are set to values less than 3, so that only privileged software can 
perform task switching. However, in multitasking applications, DPLs for some TSS descriptors may be set to 3 to 
allow task switching at the application (or user) privilege level. 


7.2.3 TSS Descriptor in 64-bit mode 


In 64-bit mode, task switching is not supported, but TSS descriptors still exist. The format of a 64-bit TSS is 
described in Section 7.7. 


In 64-bit mode, the TSS descriptor is expanded to 16 bytes (see Figure 7-4). This expansion also applies to an LDT 
descriptor in 64-bit mode. Table 3-2 provides the encoding information for the segment type field. 
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TSS (or LDT) Descriptor 
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TYPE Segment Type 


Figure 7-4. Format of TSS and LDT Descriptors in 64-bit Mode 


7.2.4 Task Register 


The task register holds the 16-bit segment selector and the entire segment descriptor (32-bit base address (64 bits 
in IA-32e mode), 16-bit segment limit, and descriptor attributes) for the TSS of the current task (see Figure 2-6). 
This information is copied from the TSS descriptor in the GDT for the current task. Figure 7-5 shows the path the 
processor uses to access the TSS (using the information in the task register). 


The task register has a visible part (that can be read and changed by software) and an invisible part (maintained 
by the processor and is inaccessible by software). The segment selector in the visible portion points to a TSS 
descriptor in the GDT. The processor uses the invisible portion of the task register to cache the segment descriptor 
for the TSS. Caching these values in a register makes execution of the task more efficient. The LTR (load task 
register) and STR (store task register) instructions load and read the visible portion of the task register: 


The LTR instruction loads a segment selector (Source operand) into the task register that points to a TSS descriptor 
in the GDT. It then loads the invisible portion of the task register with information from the TSS descriptor. LTR is a 
privileged instruction that may be executed only when the CPL is 0. It’s used during system initialization to put an 
initial value in the task register. Afterwards, the contents of the task register are changed implicitly when a task 
switch occurs. 


The STR (store task register) instruction stores the visible portion of the task register in a general-purpose register 
or memory. This instruction can be executed by code running at any privilege level in order to identify the currently 
running task. However, it is normally used only by operating system software. (If CR4.UMIP = 1, STR can be 
executed only when CPL = 0.) 


On power up or reset of the processor, segment selector and base address are set to the default value of 0; the limit 
is set to FFFFH. 


Vol. 3A 7-7 


TASK MANAGEMENT 


7.2.5 


Task 
Register 


TSS 

~< (+)< 

A 

« 
Visible Part Invisible Part 
Selector Base Address Segment Limit 

A 

GDT 
>| TSS Descriptor 


0 


Figure 7-5. Task Register 


Task-Gate Descriptor 


A task-gate descriptor provides an indirect, protected reference to a task (see Figure 7-6). It can be placed in the 
GDT, an LDT, or the IDT. The TSS segment selector field in a task-gate descriptor points to a TSS descriptor in the 
GDT. The RPL in this segment selector is not used. 


The DPL of a task-gate descriptor controls access to the TSS descriptor during a task switch. When a program or 
procedure makes a call or jump to a task through a task gate, the CPL and the RPL field of the gate selector pointing 
to the task gate must be less than or equal to the DPL of the task-gate descriptor. Note that when a task gate is 


used, the DPL of the destination TSS descriptor is not used. 
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A task can be accessed either through a task-gate descriptor or a TSS descriptor. Both of these structures satisfy 
the following needs: 


° Need for a task to have only one busy flag — Because the busy flag for a task is stored in the TSS 
descriptor, each task should have only one TSS descriptor. There may, however, be several task gates that 
reference the same TSS descriptor. 


° Need to provide selective access to tasks — Task gates fill this need, because they can reside in an LDT and 
can have a DPL that is different from the TSS descriptor's DPL. A program or procedure that does not have 
sufficient privilege to access the TSS descriptor for a task in the GDT (which usually has a DPL of 0) may be 
allowed access to the task through a task gate with a higher DPL. Task gates give the operating system greater 
latitude for limiting access to specific tasks. 


¢ Need for an interrupt or exception to be handled by an independent task — Task gates may also reside 
in the IDT, which allows interrupts and exceptions to be handled by handler tasks. When an interrupt or 
exception vector points to a task gate, the processor switches to the specified task. 


Figure 7-7 illustrates how a task gate in an LDT, a task gate in the GDT, and a task gate in the IDT can all point to 
the same task. 


LDT GDT TSS 
Task Gate 
L_y| 
Task Gate a —<d TSS Descriptor 
> 
IDT 
Task Gate 


Figure 7-7. Task Gates Referencing the Same Task 


7.3 TASK SWITCHING 


The processor transfers execution to another task in one of four cases: 
¢ The current program, task, or procedure executes a JMP or CALL instruction to a TSS descriptor in the GDT. 


¢ The current program, task, or procedure executes a JMP or CALL instruction to a task-gate descriptor in the 
GDT or the current LDT. 
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An interrupt or exception vector points to a task-gate descriptor in the IDT. 
The current task executes an IRET when the NT flag in the EFLAGS register is set. 


JMP, CALL, and IRET instructions, as well as interrupts and exceptions, are all mechanisms for redirecting a 
program. The referencing of a TSS descriptor or a task gate (when calling or jumping to a task) or the state of the 
NT flag (when executing an IRET instruction) determines whether a task switch occurs. 


The processor performs the following operations when switching to a new task: 


1. 


Obtains the TSS segment selector for the new task as the operand of the JMP or CALL instruction, from a task 
gate, or from the previous task link field (for a task switch initiated with an IRET instruction). 


Checks that the current (old) task is allowed to switch to the new task. Data-access privilege rules apply to JMP 
and CALL instructions. The CPL of the current (old) task and the RPL of the segment selector for the new task 
must be less than or equal to the DPL of the TSS descriptor or task gate being referenced. Exceptions, 
interrupts (except for those identified in the next sentence), and the IRET and INT1 instructions are permitted 
to switch tasks regardless of the DPL of the destination task-gate or TSS descriptor. For interrupts generated by 
the INT n, INT3, and INTO instructions, the DPL is checked and a general-protection exception (#GP) results if 
it is less than the CPL. 


Checks that the TSS descriptor of the new task is marked present and has a valid limit (greater than or equal 
to 67H). If the task switch was initiated by IRET and shadow stacks are enabled at the current CPL, then the 
SSP must be aligned to 8 bytes, else a #TS(current task TSS) fault is generated. If CR4.CET is 1, then the TSS 
must be a 32 bit TSS and the limit of the new task’s TSS must be greater than or equal to 107 bytes, else a 
#TS(new task TSS) fault is generated. 


Checks that the new task is available (call, jump, exception, or interrupt) or busy (IRET return). 


Checks that the current (old) TSS, new TSS, and all segment descriptors used in the task switch are paged into 
system memory. 


Saves the state of the current (old) task in the current task’s TSS. The processor finds the base address of the 
current TSS in the task register and then copies the states of the following registers into the current TSS: all the 
general-purpose registers, segment selectors from the segment registers, the temporarily saved image of the 
EFLAGS register, and the instruction pointer register (EIP). 


Loads the task register with the segment selector and descriptor for the new task's TSS. 
If CET is enabled, the processor performs following shadow stack actions: 
Read CS of new task from new task TSS 
Read EFLAGS of new task from new task TSS 
IF EFLAGS.VM = 1 
THEN 
new task CPL = 3; 
ELSE 
new task CPL = CS.RPL; 
FI; 
pushCsLipSsp = 0 
IF task switch was initiated by CALL instruction, exception or interrupt 
IF shadow stack enabled at current CPL 
IF new task CPL < CPL and current task CPL = 3 
THEN 
IA32_PL3_SSP = SSP (* user — supervisor *) 
ELSE 
pushCsLipSsp = 1 (* no privilege change; supv — supv; supv — user *) tempSSP = SSP 


1. 


The INT1 has opcode F1; the INT ninstruction with n=1 has opcode CD 01. 
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tempSsLIP =CSBASE + EIP 
tempSsCS = CS 
FI; 
FI; 
FI; 
verifyCsLIP = 0 
IF task switch was initiated by IRET 
IF shadow stacks enabled at current CPL 
IF (CPL of new Task = CPL of current Task) OR 
(CPL of new Task < 3 AND CPL of current Task < 3) OR 
(CPL or new Task < 3 AND CPL of current task = 3) 
(* no privilege change or supervisor — supervisor or user — supervisor IRET *) 
tempSsCS = shadow_stack_load 8 bytes from SSP+16; 
tempSsLIP = shadow_stack_load 8 bytes from SSP+8; 
tempSSP = shadow_stack_load 8 bytes from SSP; 
SSP = SSP + 24; 
verifyCsLIP = 1 


FI; 

// Clear busy flag on current shadow stack 

IF (SSP & 0x07 ==0) (* SSP must be aligned to 8B *) 

THEN 

expected_token_value = (SSP & ~0x07) | BUSY_BIT; (* busy - bit 0 - must be set*) 
new_token_value = SSP (* clear the busy bit *) 
shadow_stack_lock_cmpxchg8b(SSP, new_token_value, expected_token_value) 

FI; 

SSP =0 

FI; 


FI; 


9. The TSS state is loaded into the processor. This includes the LDTR register, the PDBR (control register CR3), the 
EFLAGS register, the EIP register, the general-purpose registers, and the segment selectors. A fault during the 
load of this state may corrupt architectural state. (If paging is not enabled, a PDBR value is read from the new 
task's TSS, but it is not loaded into CR3.) 


10. If the task switch was initiated with a JMP or IRET instruction, the processor clears the busy (B) flag in the 
current (old) task’s TSS descriptor; if initiated with a CALL instruction, an exception, or an interrupt: the busy 
(B) flag is left set. (See Table 7-2.) 


11. If the task switch was initiated with an IRET instruction, the processor clears the NT flag in a temporarily saved 
image of the EFLAGS register; if initiated with a CALL or JMP instruction, an exception, or an interrupt, the NT 
flag is left unchanged in the saved EFLAGS image. 


12. If the task switch was initiated with a CALL instruction, an exception, or an interrupt, the processor will set the 
NT flag in the EFLAGS loaded from the new task. If initiated with an IRET instruction or JMP instruction, the NT 
flag will reflect the state of NT in the EFLAGS loaded from the new task (see Table 7-2). 


13. If the task switch was initiated with a CALL instruction, JMP instruction, an exception, or an interrupt, the 
processor sets the busy (B) flag in the new task’s TSS descriptor; if initiated with an IRET instruction, the busy 
(B) flag is left set. 


14. The descriptors associated with the segment selectors are loaded and qualified. Any errors associated with this 
loading and qualification occur in the context of the new task and may corrupt architectural state. 
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15. If CET is enabled, the processor performs following shadow stack actions: 
IF shadow stack enabled at current CPL OR indirect branch tracking at current CPL 
THEN 
IF EFLAGS.VM = 1 
THEN #TSS(new-Task-TSS); FI; 
FI; 
IF shadow stack enabled at current CPL 
IF task switch initiated by CALL instruction, JMP instruction, interrupt or exception (* switch stack *) 
new_SSP <— Load the 4 byte from offset 104 in the TSS 
// Verify new SSP to be legal 
IF new_SSP & 0x07 != 0 
THEN #TSS(New-Task-TSS); FI; 
expected_token_value = SSP; (* busy - bit 0 - must be clear *) 
new_token_value = SSP | BUSY_BIT (* set the busy bit - bit 0*) 
IF shadow_stack_lock_cmpxchg8b(SSP, new_token_value, 
expected_token_value) != expected_token_value 
THEN #TSS(New-Task-TSS); FI; 
SSP = new_SSP 
IF pushCsLipSsp = 1 (* call, int, exception from user — user or supv — supv or supv — user *) 
Push tempSsCsS, tempSsLip, tempSsSSP on shadow stack using 8B pushes 
FI; 
FI; 
FI; 
IF task switch initiated by IRET 
IF verifyCsLIP = 1 
(* do 64 bit comparisons; CS zero padded to 64 bit; CSBASE+EIP zero padded to 64 bit *) 
IF tempSsCS and tempSsLIP do not match CS and CSBASE+EIP 
THEN #CP(FAR-RET/IRET); FI; 
FI; 
IF ShadowStackEnabled(CPL) 
THEN 
IF (verifyCsLIP == 0) tempSSP = IA32_PL3_SSP; 
IF tempSSP & 0x03 != 0 THEN #CP(FAR-RET/IRET) // verify aligned to 4 bytes 
IF tempSSP[63:32] != 0 THEN # CP(FAR-RET/IRET) 
SSP = tempSSP 
FI; 
FI; 
IF EndbranchEnabled(CPL) 
IF task switch initiated by CALL instruction, JMP instruction, interrupt or exception 
IF CPL = 3 
THEN 
TA32_U_CET.TRACKER = WAIT_FOR_ENDBRANCH 
IA32_U_CET.SUPPRESS = 0 
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ELSE 
IA32_S_CET.TRACKER = WAIT_FOR_ENDBRANCH 
IA32_S_CET.SUPPRESS = 0 
FI; 
FI; 
FI; 


16. Begins executing the new task. (To an exception handler, the first instruction of the new task appears not to 
have been executed.) 


NOTES 


If all checks and saves have been carried out successfully, the processor commits to the task 
switch. If an unrecoverable error occurs in steps 1 through 8, the processor does not complete the 
task switch and ensures that the processor is returned to its state prior to the execution of the 
instruction that initiated the task switch. 


If an unrecoverable error occurs in step 9, architectural state may be corrupted, but an attempt will 
be made to handle the error in the prior execution environment. If an unrecoverable error occurs 
after the commit point (in step 13), the processor completes the task switch (without performing 
additional access and segment availability checks) and generates the appropriate exception prior to 
beginning execution of the new task. 


If exceptions occur after the commit point, the exception handler must finish the task switch itself 
before allowing the processor to begin executing the new task. See Chapter 6, “Interrupt 
10—Invalid TSS Exception (#TS),” for more information about the affect of exceptions on a task 
when they occur after the commit point of a task switch. 


The state of the currently executing task is always saved when a successful task switch occurs. If the task is 
resumed, execution starts with the instruction pointed to by the saved EIP value, and the registers are restored to 
the values they held when the task was suspended. 


When switching tasks, the privilege level of the new task does not inherit its privilege level from the suspended 
task. The new task begins executing at the privilege level specified in the CPL field of the CS register, which is 
loaded from the TSS. Because tasks are isolated by their separate address spaces and TSSs and because privilege 
rules control access to a TSS, software does not need to perform explicit privilege checks on a task switch. 


Table 7-1 shows the exception conditions that the processor checks for when switching tasks. It also shows the 
exception that is generated for each check if an error is detected and the segment that the error code references. 
(The order of the checks in the table is the order used in the P6 family processors. The exact order is model specific 
and may be different for other IA-32 processors.) Exception handlers designed to handle these exceptions may be 
subject to recursive calls if they attempt to reload the segment selector that generated the exception. The cause of 
the exception (or the first of multiple causes) should be fixed before reloading the selector. 


Table 7-1. Exception Conditions Checked During a Task Switch 


Condition Checked Exception! Error Code Reference? 
Segment selector for a TSS descriptor references #GP New Task’s TSS 
the GDT and is within the limits of the table. #TS (for IRET) 
P bit is set in TSS descriptor. #NP New Task’s TSS 
TSS descriptor is not busy (for task switch initiated by a call, interrupt, or #GP (for JMP, CALL, INT) | Task’s back-link TSS 
exception). 
TSS descriptor is not busy (for task switch initiated by an IRET instruction). #TS (for IRET) New Task’s TSS 
TSS segment limit greater than or equal to 108 (for 32-bit TSS) or 44 (for 16-bit | #TS New Task's TSS 
TSS). 
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Table 7-1. Exception Conditions Checked During a Task Switch (Contd.) 


Condition Checked Exception’ Error Code Reference 
TSS segment limit greater than or equal to 108 (for 32-bit TSS) if CR4.CET = 1.2 | #TS New Task’s TSS 
If shadow stack enabled and SSP not aligned to 8 bytes (for task switch initiated | #TS Current Task’s TSS 


by an IRET instruction). 


Registers are loaded from the values in the TSS. 


LDT segment selector of new task is valid 4. #TS New Task’s LDT 

If code segment is non-conforming, its DPL should equal its RPL. #TS New Code Segment 
If code segment is conforming, its DPL should be less than or equal to its RPL. #TS New Code Segment 
SS segment selector is valid @. #TS New Stack Segment 
P bit is set in stack segment descriptor. #SS New Stack Segment 
Stack segment DPL should equal CPL. #TS New stack segment 
P bit is set in new task’s LDT descriptor. #TS New Task's LDT 

CS segment selector is valid 4. #TS New Code Segment 
P bit is set in code segment descriptor. #NP New Code Segment 
Stack segment DPL should equal its RPL. #TS New Stack Segment 
DS, ES, FS, and GS segment selectors are valid 4. #TS New Data Segment 
DS, ES, FS, and GS segments are readable. #TS New Data Segment 
P bits are set in descriptors of DS, ES, FS, and GS segments. #NP New Data Segment 
DS, ES, FS, and GS segment DPL greater than or equal to CPL (unless these are #TS New Data Segment 
conforming segments). 

Shadow Stack Pointer in a task not aligned to 8 bytes (for task switch initiated by | #TS New Task’s TSS 

a call, interrupt, or exception).? 

If EFLAGS.VM=1 and shadow stacks are enabled. #TS New Task’s TSS 
Supervisor Shadow Stack Token verification failures (for task switch initiated by a | #TS New Task’s TSS 


call, interrupt, jump, or exception):3 
- Busy bit already set. 
- Address in Shadow stack token does not match SSP value from TSS. 


If task switch initiated by IRET, CS and LIP stored on old task shadow stack does | #CP FAR-RET/IRET 
not match CS and LIP of new task. 
If task switch initiated by IRET and SSP of new task loaded from shadow stack of | #CP FAR-RET/IRET 


old task (if new task CPL is < 3) OR the SSP from IA32_PL3_SSP (if new task CPL 
= 3) fails the following checks:3 


- Not aligned to 4 bytes. 
- Is beyond 4G. 


NOTES: 

1. #NP is segment-not-present exception, #GP is general-protection exception, #TS is invalid-TSS exception, and #SS is stack-fault 
exception. 

2. The error code contains an index to the segment descriptor referenced in this column. 

3. Valid when CET is enabled. 

4. A segment selector is valid if it is in a compatible type of table (GDT or LDT), occupies an address within the table’s segment limit, 
and refers to a compatible type of descriptor (for example, a segment selector in the CS register only is valid when it points to a 
code-segment descriptor). 


The TS (task switched) flag in the control register CRO is set every time a task switch occurs. System software uses 
the TS flag to coordinate the actions of floating-point unit when generating floating-point exceptions with the rest 
of the processor. The TS flag indicates that the context of the floating-point unit may be different from that of the 
current task. See Section 2.5, “Control Registers”, for a detailed description of the function and use of the TS flag. 
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7.4 TASK LINKING 


The previous task link field of the TSS (sometimes called the “backlink”) and the NT flag in the EFLAGS register are 
used to return execution to the previous task. EFLAGS.NT = 1 indicates that the currently executing task is nested 
within the execution of another task. 


When a CALL instruction, an interrupt, or an exception causes a task switch: the processor copies the segment 
selector for the current TSS to the previous task link field of the TSS for the new task; it then sets EFLAGS.NT = 1. 
If software uses an IRET instruction to suspend the new task, the processor checks for EFLAGS.NT = 1; it then 
uses the value in the previous task link field to return to the previous task. See Figures 7-8. 


When a JMP instruction causes a task switch, the new task is not nested. The previous task link field is not used and 
EFLAGS.NT = 0. Use a JMP instruction to dispatch a new task when nesting is not desired. 


Top Level Nested More Deeply Currently Executing 
Task Task Nested Task Task 
TSS TSS TSS EFLAGS 
NT=1 
NT=0 NT=1 NT=1 
Previous Previous Previous 
Task Link Task Link Task Link Task Register 


Figure 7-8. Nested Tasks 


Table 7-2 shows the busy flag (in the TSS segment descriptor), the NT flag, the previous task link field, and TS flag 
(in control register CRO) during a task switch. 


The NT flag may be modified by software executing at any privilege level. It is possible for a program to set the NT 
flag and execute an IRET instruction. This might randomly invoke the task specified in the previous link field of the 
current task's TSS. To keep such spurious task switches from succeeding, the operating system should initialize the 
previous task link field in every TSS that it creates to 0. 


Table 7-2. Effect of a Task Switch on Busy Flag, NT Flag, Previous Task Link Field, and TS Flag 


NT flag of new task. 


NT flag of old task. 


Previous task link field of new 
task. 


Previous task link field of old 
task. 


TS flag in control register CRO. 


Set to value from TSS of new 
task. 


No change. 


No change. 


No change. 


Flag is set. 


Flag or Field Effect of JMP instruction Effect of CALL Instruction or Effect of IRET 
Interrupt Instruction 
Busy (B) flag of new task. Flag is set. Must have been Flag is set. Must have been No change. Must have been set. 
clear before. clear before. 
Busy flag of old task. Flag is cleared. No change. Flag is currently Flag is cleared. 


set. 


Flag is set. 


No change. 


Loaded with selector 
for old task’s TSS. 


No change. 


Flag is set. 


Set to value from TSS of new 
task. 


Flag is cleared. 


No change. 


No change. 


Flag is set. 
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7.4.1 Use of Busy Flag To Prevent Recursive Task Switching 


A TSS allows only one context to be saved for a task; therefore, once a task is called (dispatched), a recursive (or 
re-entrant) call to the task would cause the current state of the task to be lost. The busy flag in the TSS segment 
descriptor is provided to prevent re-entrant task switching and a subsequent loss of task state information. The 
processor manages the busy flag as follows: 


1. When dispatching a task, the processor sets the busy flag of the new task. 


2. If during a task switch, the current task is placed in a nested chain (the task switch is being generated by a 
CALL instruction, an interrupt, or an exception), the busy flag for the current task remains set. 


3. When switching to the new task (initiated by a CALL instruction, interrupt, or exception), the processor 
generates a general-protection exception (#GP) if the busy flag of the new task is already set. If the task switch 
is initiated with an IRET instruction, the exception is not raised because the processor expects the busy flag to 
be set. 


4. When a task is terminated by a jump to a new task (initiated with a JMP instruction in the task code) or by an 
IRET instruction in the task code, the processor clears the busy flag, returning the task to the “not busy” state. 


The processor prevents recursive task switching by preventing a task from switching to itself or to any task ina 
nested chain of tasks. The chain of nested suspended tasks may grow to any length, due to multiple calls, inter- 
rupts, or exceptions. The busy flag prevents a task from being invoked if it is in this chain. 


The busy flag may be used in multiprocessor configurations, because the processor follows a LOCK protocol (on the 
bus or in the cache) when it sets or clears the busy flag. This lock keeps two processors from invoking the same 
task at the same time. See Section 8.1.2.1, “Automatic Locking,” for more information about setting the busy flag 
in a multiprocessor applications. 


7.4.2 Modifying Task Linkages 


In a uniprocessor system, in situations where it is necessary to remove a task from a chain of linked tasks, use the 
following procedure to remove the task: 


1. Disable interrupts. 


2. Change the previous task link field in the TSS of the pre-empting task (the task that suspended the task to be 
removed). It is assumed that the pre-empting task is the next task (newer task) in the chain from the task to 
be removed. Change the previous task link field to point to the TSS of the next oldest task in the chain or to an 
even older task in the chain. 


3. Clear the busy (B) flag in the TSS segment descriptor for the task being removed from the chain. If more than 
one task is being removed from the chain, the busy flag for each task being remove must be cleared. 


4. Enable interrupts. 


In a multiprocessing system, additional synchronization and serialization operations must be added to this proce- 
dure to ensure that the TSS and its segment descriptor are both locked when the previous task link field is changed 
and the busy flag is cleared. 


7.5 TASK ADDRESS SPACE 


The address space for a task consists of the segments that the task can access. These segments include the code, 
data, stack, and system segments referenced in the TSS and any other segments accessed by the task code. The 
segments are mapped into the processor’s linear address space, which is in turn mapped into the processor's phys- 
ical address space (either directly or through paging). 


The LDT segment field in the TSS can be used to give each task its own LDT. Giving a task its own LDT allows the 
task address space to be isolated from other tasks by placing the segment descriptors for all the segments associ- 
ated with the task in the task’s LDT. 


It also is possible for several tasks to use the same LDT. This is a memory-efficient way to allow specific tasks to 
communicate with or control each other, without dropping the protection barriers for the entire system. 


7-16 Vol. 3A 


TASK MANAGEMENT 


Because all tasks have access to the GDT, it also is possible to create shared segments accessed through segment 
descriptors in this table. 


If paging is enabled, the CR3 register (PDBR) field in the TSS allows each task to have its own set of page tables 
for mapping linear addresses to physical addresses. Or, several tasks can share the same set of page tables. 


7.5.1 Mapping Tasks to the Linear and Physical Address Spaces 


Tasks can be mapped to the linear address space and physical address space in one of two ways: 


¢ One linear-to-physical address space mapping is shared among all tasks. — When paging is not 
enabled, this is the only choice. Without paging, all linear addresses map to the same physical addresses. When 
paging is enabled, this form of linear-to-physical address space mapping is obtained by using one page 
directory for all tasks. The linear address space may exceed the available physical space if demand-paged 
virtual memory is supported. 


° Each task has its own linear address space that is mapped to the physical address space. — This form 
of mapping is accomplished by using a different page directory for each task. Because the PDBR (control 
register CR3) is loaded on task switches, each task may have a different page directory. 


The linear address spaces of different tasks may map to completely distinct physical addresses. If the entries of 
different page directories point to different page tables and the page tables point to different pages of physical 
memory, then the tasks do not share physical addresses. 


With either method of mapping task linear address spaces, the TSSs for all tasks must lie in a shared area of the 
physical space, which is accessible to all tasks. This mapping is required so that the mapping of TSS addresses does 
not change while the processor is reading and updating the TSSs during a task switch. The linear address space 
mapped by the GDT also should be mapped to a shared area of the physical space; otherwise, the purpose of the 
GDT is defeated. Figure 7-9 shows how the linear address spaces of two tasks can overlap in the physical space by 
sharing page tables. 


TSS Page Directories Page Tables Page Frames 
Task A 
Task A TSS —> 
PTE Task A 
PTE > 
PDBR [| PDE } | PTE TaskA 
PDE 
Shared PT 
Shared 
PTE a 
— S| PTE Shared 
Task B TSS 
> 
Task B 
PDBR [| PDE PTE 
PDE [a PTE Task B 
> 


Figure 7-9. Overlapping Linear-to-Physical Mappings 
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7.5.2 Task Logical Address Space 


To allow the sharing of data among tasks, use the following techniques to create shared logical-to-physical 
address-space mappings for data segments: 


¢ Through the segment descriptors in the GDT — All tasks must have access to the segment descriptors in 
the GDT. If some segment descriptors in the GDT point to segments in the linear-address space that are 
mapped into an area of the physical-address space common to all tasks, then all tasks can share the data and 
code in those segments. 


¢ Through a shared LDT — Two or more tasks can use the same LDT if the LDT fields in their TSSs point to the 
same LDT. If some segment descriptors in a shared LDT point to segments that are mapped to a common area 
of the physical address space, the data and code in those segments can be shared among the tasks that share 
the LDT. This method of sharing is more selective than sharing through the GDT, because the sharing can be 
limited to specific tasks. Other tasks in the system may have different LDTs that do not give them access to the 
shared segments. 


* Through segment descriptors in distinct LDTs that are mapped to common addresses in linear 
address space — If this common area of the linear address space is mapped to the same area of the physical 
address space for each task, these segment descriptors permit the tasks to share segments. Such segment 
descriptors are commonly called aliases. This method of sharing is even more selective than those listed above, 
because, other segment descriptors in the LDTs may point to independent linear addresses which are not 
shared. 


7.6 16-BIT TASK-STATE SEGMENT (TSS) 


The 32-bit IA-32 processors also recognize a 16-bit TSS format like the one used in Intel 286 processors (see 
Figure 7-10). This format is supported for compatibility with software written to run on earlier IA-32 processors. 


The following information is important to know about the 16-bit TSS. 
* Do not use a 16-bit TSS to implement a virtual-8086 task. 
¢ The valid segment limit for a 16-bit TSS is 2CH. 


¢ The 16-bit TSS does not contain a field for the base address of the page directory, which is loaded into control 
register CR3. A separate set of page tables for each task is not supported for 16-bit tasks. If a 16-bit task is 
dispatched, the page-table structure for the previous task is used. 


¢ The I/O base address is not included in the 16-bit TSS. None of the functions of the I/O map are supported. 
¢ When task state is saved in a 16-bit TSS, the upper 16 bits of the EFLAGS register and the EIP register are lost. 


¢ When the general-purpose registers are loaded or saved from a 16-bit TSS, the upper 16 bits of the registers 
are modified and not maintained. 
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15 0 
Task LDT Selector 42 
DS Selector 40 
SS Selector 38 
CS Selector 36 
ES Selector 34 
DI 32 
SI 30 
BP 28 
SP 26 
BX 24 
DX 22 
CX 20 
AX 18 
FLAG Word 16 
IP (Entry Point) 14 
SS2 12 
SP2 10 
SS1 8 
SP1 6 
Sso 4 
SPO 2 
Previous Task Link 0 


Figure 7-10. 16-Bit TSS Format 


7.7 TASK MANAGEMENT IN 64-BIT MODE 


In 64-bit mode, task structure and task state are similar to those in protected mode. However, the task switching 
mechanism available in protected mode is not supported in 64-bit mode. Task management and switching must be 
performed by software. The processor issues a general-protection exception (#GP) if the following is attempted in 
64-bit mode: 


¢ Control transfer to a TSS or a task gate using JMP, CALL, INT n, INT3, INTO, INT1, or interrupt. 
¢ An IRET with EFLAGS.NT (nested task) set to 1. 


Although hardware task-switching is not supported in 64-bit mode, a 64-bit task state segment (TSS) must exist. 
Figure 7-11 shows the format of a 64-bit TSS. The TSS holds information important to 64-bit mode and that is not 
directly related to the task-switch mechanism. This information includes: 


° RSPn — The full 64-bit canonical forms of the stack pointers (RSP) for privilege levels 0-2. 
¢ ISTn — The full 64-bit canonical forms of the interrupt stack table (IST) pointers. 
¢ I/O map base address — The 16-bit offset to the I/O permission bit map from the 64-bit TSS base. 


The operating system must create at least one 64-bit TSS after activating IA-32e mode. It must execute the LTR 
instruction (in 64-bit mode) to load the TR register with a pointer to the 64-bit TSS responsible for both 64-bit- 
mode programs and compatibility-mode programs. 
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I/O Map Base Address Reserved 100 
Reserved 96 
Reserved 92 
IST7 (upper 32 bits) 88 
IST7 (lower 32 bits) 84 
IST6 (upper 32 bits) 80 
IST6 (lower 32 bits) 76 
IST5 (upper 32 bits) 72 
IST5 (lower 32 bits) 68 
IST4 (upper 32 bits) 64 
IST4 (lower 32 bits) 60 
IST3 (upper 32 bits) 56 
IST3 (lower 32 bits) 52 
IST2 (upper 32 bits) 48 
IST2 (lower 32 bits) 44 
IST1 (upper 32 bits) 40 
IST1 (lower 32 bits) 36 
Reserved 32 


Reserved 28 


RSP2 (upper 32 bits) 24 
RSP2 (lower 32 bits) 20 
RSP1 (upper 32 bits) 16 
RSP1 (lower 32 bits) 12 
RSPO (upper 32 bits) 8 
RSPO (lower 32 bits) 4 
Reserved 0 


Reserved bits. Set to 0. 


Figure 7-11. 64-Bit TSS Format 
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MULTIPLE-PROCESSOR MANAGEMENT 


The Intel 64 and IA-32 architectures provide mechanisms for managing and improving the performance of multiple 
processors connected to the same system bus. These include: 


* Bus locking and/or cache coherency management for performing atomic operations on system memory. 
°®  Serializing instructions. 


¢ An advance programmable interrupt controller (APIC) located on the processor chip (see Chapter 10, 
“Advanced Programmable Interrupt Controller (APIC)”). This feature was introduced by the Pentium processor. 


¢ A second-level cache (level 2, L2). For the Pentium 4, Intel Xeon, and P6 family processors, the L2 cache is 
included in the processor package and is tightly coupled to the processor. For the Pentium and Intel486 
processors, pins are provided to support an external L2 cache. 


¢ A third-level cache (level 3, L3). For Intel Xeon processors, the L3 cache is included in the processor package 
and is tightly coupled to the processor. 


¢ Intel Hyper-Threading Technology. This extension to the Intel 64 and IA-32 architectures enables a single 
processor core to execute two or more threads concurrently (see Section 8.5, “Intel® Hyper-Threading 
Technology and Intel® Multi-Core Technology”). 


These mechanisms are particularly useful in symmetric-multiprocessing (SMP) systems. However, they can also be 
used when an Intel 64 or IA-32 processor and a special-purpose processor (such as a communications, graphics, 
or video processor) share the system bus. 


These multiprocessing mechanisms have the following characteristics: 


* To maintain system memory coherency — When two or more processors are attempting simultaneously to 
access the same address in system memory, some communication mechanism or memory access protocol 
must be available to promote data coherency and, in some instances, to allow one processor to temporarily lock 
a memory location. 

* To maintain cache consistency — When one processor accesses data cached on another processor, it must not 
receive incorrect data. If it modifies data, all other processors that access that data must receive the modified 
data. 

° To allow predictable ordering of writes to memory — In some circumstances, it is important that memory writes 
be observed externally in precisely the same order as programmed. 

°® To distribute interrupt handling among a group of processors — When several processors are operating ina 
system in parallel, it is useful to have a centralized mechanism for receiving interrupts and distributing them to 
available processors for servicing. 

° To increase system performance by exploiting the multi-threaded and multi-process nature of contemporary 
operating systems and applications. 

The caching mechanism and cache consistency of Intel 64 and IA-32 processors are discussed in Chapter 11. The 

APIC architecture is described in Chapter 10. Bus and memory locking, serializing instructions, memory ordering, 

and Intel Hyper-Threading Technology are discussed in the following sections. 


8.1 LOCKED ATOMIC OPERATIONS 


The 32-bit IA-32 processors support locked atomic operations on locations in system memory. These operations 
are typically used to manage shared data structures (such as semaphores, segment descriptors, system segments, 
or page tables) in which two or more processors may try simultaneously to modify the same field or flag. The 
processor uses three interdependent mechanisms for carrying out locked atomic operations: 


* Guaranteed atomic operations 
* Bus locking, using the LOCK# signal and the LOCK instruction prefix 
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® Cache coherency protocols that ensure that atomic operations can be carried out on cached data structures 
(cache lock); this mechanism is present in the Pentium 4, Intel Xeon, and P6 family processors 


These mechanisms are interdependent in the following ways. Certain basic memory transactions (such as reading 
or writing a byte in system memory) are always guaranteed to be handled atomically. That is, once started, the 
processor guarantees that the operation will be completed before another processor or bus agent is allowed access 
to the memory location. The processor also supports bus locking for performing selected memory operations (such 
as a read-modify-write operation in a shared area of memory) that typically need to be handled atomically, but are 
not automatically handled this way. Because frequently used memory locations are often cached in a processor’s L1 
or L2 caches, atomic operations can often be carried out inside a processor’s caches without asserting the bus lock. 
Here the processor’s cache coherency protocols ensure that other processors that are caching the same memory 
locations are managed properly while atomic operations are performed on cached memory locations. 


NOTE 


Where there are contested lock accesses, software may need to implement algorithms that ensure 
fair access to resources in order to prevent lock starvation. The hardware provides no resource that 
guarantees fairness to participating agents. It is the responsibility of software to manage the 
fairness of semaphores and exclusive locking functions. 


The mechanisms for handling locked atomic operations have evolved with the complexity of IA-32 processors. More 
recent IA-32 processors (such as the Pentium 4, Intel Xeon, and P6 family processors) and Intel 64 provide a more 
refined locking mechanism than earlier processors. These mechanisms are described in the following sections. 


8.1.1 Guaranteed Atomic Operations 


The Intel486 processor (and newer processors since) guarantees that the following basic memory operations will 
always be carried out atomically: 


® Reading or writing a byte 
¢ Reading or writing a word aligned on a 16-bit boundary 
¢ Reading or writing a doubleword aligned on a 32-bit boundary 


The Pentium processor (and newer processors since) guarantees that the following additional memory operations 
will always be carried out atomically: 


* Reading or writing a quadword aligned on a 64-bit boundary 
¢ 16-bit accesses to uncached memory locations that fit within a 32-bit data bus 


The P6 family processors (and newer processors since) guarantee that the following additional memory operation 
will always be carried out atomically: 


¢ Unaligned 16-, 32-, and 64-bit accesses to cached memory that fit within a cache line 


Accesses to cacheable memory that are split across cache lines and page boundaries are not guaranteed to be 
atomic by the Intel Core 2 Duo, Intel® Atom™, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, P6 family, 
Pentium, and Intel486 processors. The Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium M, Pentium 4, Intel 
Xeon, and P6 family processors provide bus control signals that permit external memory subsystems to make split 
accesses atomic; however, nonaligned data accesses will seriously impact the performance of the processor and 
should be avoided. 


An x87 instruction or an SSE instructions that accesses data larger than a quadword may be implemented using 
multiple memory accesses. If such an instruction stores to memory, some of the accesses may complete (writing 
to memory) while another causes the operation to fault for architectural reasons (e.g. due an page-table entry that 
is marked “not present”). In this case, the effects of the completed accesses may be visible to software even 
though the overall instruction caused a fault. If TLB invalidation has been delayed (see Section 4.10.4.4), such 
page faults may occur even if all accesses are to the same page. 
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8.1.2 Bus Locking 


Intel 64 and IA-32 processors provide a LOCK# signal that is asserted automatically during certain critical memory 
operations to lock the system bus or equivalent link. While this output signal is asserted, requests from other 
processors or bus agents for control of the bus are blocked. Software can specify other occasions when the LOCK 
semantics are to be followed by prepending the LOCK prefix to an instruction. 


In the case of the Intel386, Intel486, and Pentium processors, explicitly locked instructions will result in the asser- 
tion of the LOCK# signal. It is the responsibility of the hardware designer to make the LOCK# signal available in 
system hardware to control memory accesses among processors. 


For the P6 and more recent processor families, if the memory area being accessed is cached internally in the 
processor, the LOCK# signal is generally not asserted; instead, locking is only applied to the processor's caches 
(see Section 8.1.4, “Effects of a LOCK Operation on Internal Processor Caches”). 


8.1.2.1 Automatic Locking 


The operations on which the processor automatically follows the LOCK semantics are as follows: 
¢ When executing an XCHG instruction that references memory. 


¢ When setting the B (busy) flag of a TSS descriptor — The processor tests and sets the busy flag in the 
type field of the TSS descriptor when switching to a task. To ensure that two processors do not switch to the 
same task simultaneously, the processor follows the LOCK semantics while testing and setting this flag. 


¢ When updating segment descriptors — When loading a segment descriptor, the processor will set the 
accessed flag in the segment descriptor if the flag is clear. During this operation, the processor follows the LOCK 
semantics so that the descriptor will not be modified by another processor while it is being updated. For this 
action to be effective, operating-system procedures that update descriptors should use the following steps: 


— Use a locked operation to modify the access-rights byte to indicate that the segment descriptor is not- 
present, and specify a value for the type field that indicates that the descriptor is being updated. 


— Update the fields of the segment descriptor. (This operation may require several memory accesses; 
therefore, locked operations cannot be used.) 


— Use a locked operation to modify the access-rights byte to indicate that the segment descriptor is valid and 
present. 


¢ The Intel386 processor always updates the accessed flag in the segment descriptor, whether it is clear or not. 
The Pentium 4, Intel Xeon, P6 family, Pentium, and Intel486 processors only update this flag if it is not already 
set. 


¢ When updating page-directory and page-table entries — When updating page-directory and page-table 
entries, the processor uses locked cycles to set the accessed and dirty flag in the page-directory and page-table 
entries. 


¢ Acknowledging interrupts — After an interrupt request, an interrupt controller may use the data bus to send 
the interrupt’s vector to the processor. The processor follows the LOCK semantics during this time to ensure 
that no other data appears on the data bus while the vector is being transmitted. 


8.1.2.2 Software Controlled Bus Locking 


To explicitly force the LOCK semantics, software can use the LOCK prefix with the following instructions when they 
are used to modify a memory location. An invalid-opcode exception (#UD) is generated when the LOCK prefix is 
used with any other instruction or when no write operation is made to memory (that is, when the destination 
operand is in a register). 


° The bit test and modify instructions (BTS, BTR, and BTC). 

° The exchange instructions (XADD, CMPXCHG, and CMPXCHG8B). 

¢ The LOCK prefix is automatically assumed for XCHG instruction. 

¢ The following single-operand arithmetic and logical instructions: INC, DEC, NOT, and NEG. 

¢ ~The following two-operand arithmetic and logical instructions: ADD, ADC, SUB, SBB, AND, OR, and XOR. 
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A locked instruction is guaranteed to lock only the area of memory defined by the destination operand, but may be 
interpreted by the system as a lock for a larger memory area. 


Software should access semaphores (shared memory used for signalling between multiple processors) using iden- 
tical addresses and operand lengths. For example, if one processor accesses a semaphore using a word access, 
other processors should not access the semaphore using a byte access. 


NOTE 


Do not implement semaphores using the WC memory type. Do not perform non-temporal stores to 
a cache line containing a location used to implement a semaphore. 


The integrity of a bus lock is not affected by the alignment of the memory field. The LOCK semantics are followed 
for as many bus cycles as necessary to update the entire operand. However, it is recommend that locked accesses 
be aligned on their natural boundaries for better system performance: 


¢ Any boundary for an 8-bit access (locked or otherwise). 
° 16-bit boundary for locked word accesses. 

® 32-bit boundary for locked doubleword accesses. 

° 64-bit boundary for locked quadword accesses. 


Locked operations are atomic with respect to all other memory operations and all externally visible events. Only 
instruction fetch and page table accesses can pass locked instructions. Locked instructions can be used to synchro- 
nize data written by one processor and read by another processor. 


For the P6 family processors, locked operations serialize all outstanding load and store operations (that is, wait for 
them to complete). This rule is also true for the Pentium 4 and Intel Xeon processors, with one exception. Load 
operations that reference weakly ordered memory types (such as the WC memory type) may not be serialized. 


Locked instructions should not be used to ensure that data written can be fetched as instructions. 


NOTE 


The locked instructions for the current versions of the Pentium 4, Intel Xeon, P6 family, Pentium, 
and Intel486 processors allow data written to be fetched as instructions. However, Intel 
recommends that developers who require the use of self-modifying code use a different synchro- 
nizing mechanism, described in the following sections. 


8.1.3 Handling Self- and Cross-Modifying Code 


The act of a processor writing data into a currently executing code segment with the intent of executing that data 
as code is called self-modifying code. IA-32 processors exhibit model-specific behavior when executing self- 
modified code, depending upon how far ahead of the current execution pointer the code has been modified. 


As processor microarchitectures become more complex and start to speculatively execute code ahead of the retire- 
ment point (as in P6 and more recent processor families), the rules regarding which code should execute, pre- or 
post-modification, become blurred. To write self-modifying code and ensure that it is compliant with current and 
future versions of the IA-32 architectures, use one of the following coding options: 


(* OPTION 1 *) 

Store modified code (as data) into code segment; 
Jump to new code or an intermediate location; 
Execute new code; 


(* OPTION 2 *) 

Store modified code (as data) into code segment; 

Execute a Serializing instruction; (* For example, CPUID instruction *) 
Execute new code; 


The use of one of these options is not required for programs intended to run on the Pentium or Intel486 processors, 
but are recommended to ensure compatibility with the P6 and more recent processor families. 
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Self-modifying code will execute at a lower level of performance than non-self-modifying or normal code. The 
degree of the performance deterioration will depend upon the frequency of modification and specific characteristics 
of the code. 


The act of one processor writing data into the currently executing code segment of a second processor with the 
intent of having the second processor execute that data as code is called cross-modifying code. As with self- 
modifying code, IA-32 processors exhibit model-specific behavior when executing cross-modifying code, 
depending upon how far ahead of the executing processors current execution pointer the code has been modified. 


To write cross-modifying code and ensure that it is compliant with current and future versions of the IA-32 archi- 
tecture, the following processor synchronization algorithm must be implemented: 


(* Action of Modifying Processor *) 

Memory_Flag := 0; (* Set Memory_Flag to value other than 1 *) 
Store modified code (as data) into code segment; 
Memory_Flag := 1; 


(* Action of Executing Processor *) 
WHILE (Memory_Flag # 1) 
Wait for code to update; 
ELIHW; 
Execute serializing instruction; (* For example, CPUID instruction *) 
Begin executing modified code; 


(The use of this option is not required for programs intended to run on the Intel486 processor, but is recommended 
to ensure compatibility with the Pentium 4, Intel Xeon, P6 family, and Pentium processors.) 


Like self-modifying code, cross-modifying code will execute at a lower level of performance than non-cross-modi- 
fying (normal) code, depending upon the frequency of modification and specific characteristics of the code. 


The restrictions on self-modifying code and cross-modifying code also apply to the Intel 64 architecture. 


8.1.4 Effects of a LOCK Operation on Internal Processor Caches 


For the Intel486 and Pentium processors, the LOCK# signal is always asserted on the bus during a LOCK operation, 
even if the area of memory being locked is cached in the processor. 


For the P6 and more recent processor families, if the area of memory being locked during a LOCK operation is 
cached in the processor that is performing the LOCK operation as write-back memory and is completely contained 
in a cache line, the processor may not assert the LOCK# signal on the bus. Instead, it will modify the memory loca- 
tion internally and allow it’s cache coherency mechanism to ensure that the operation is carried out atomically. This 
operation is called “cache locking.” The cache coherency mechanism automatically prevents two or more proces- 
sors that have cached the same area of memory from simultaneously modifying data in that area. 


8.2 MEMORY ORDERING 


The term memory ordering refers to the order in which the processor issues reads (loads) and writes (stores) 
through the system bus to system memory. The Intel 64 and IA-32 architectures support several memory-ordering 
models depending on the implementation of the architecture. For example, the Intel386 processor enforces 
program ordering (generally referred to as strong ordering), where reads and writes are issued on the system 
bus in the order they occur in the instruction stream under all circumstances. 


To allow performance optimization of instruction execution, the IA-32 architecture allows departures from strong- 
ordering model called processor ordering in Pentium 4, Intel Xeon, and P6 family processors. These processor- 
ordering variations (called here the memory-ordering model) allow performance enhancing operations such as 
allowing reads to go ahead of buffered writes. The goal of any of these variations is to increase instruction execu- 
tion speeds, while maintaining memory coherency, even in multiple-processor systems. 


Section 8.2.1 and Section 8.2.2 describe the memory-ordering implemented by Intel486, Pentium, Intel Core 2 
Duo, Intel Atom, Intel Core Duo, Pentium 4, Intel Xeon, and P6 family processors. Section 8.2.3 gives examples 
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illustrating the behavior of the memory-ordering model on IA-32 and Intel-64 processors. Section 8.2.4 considers 
the special treatment of stores for string operations and Section 8.2.5 discusses how memory-ordering behavior 
may be modified through the use of specific instructions. 


8.2.1 Memory Ordering in the Intel® Pentium® and Intel486"" Processors 


The Pentium and Intel486 processors follow the processor-ordered memory model; however, they operate as 
strongly-ordered processors under most circumstances. Reads and writes always appear in programmed order at 
the system bus—except for the following situation where processor ordering is exhibited. Read misses are 
permitted to go ahead of buffered writes on the system bus when all the buffered writes are cache hits and, there- 
fore, are not directed to the same address being accessed by the read miss. 


In the case of I/O operations, both reads and writes always appear in programmed order. 


Software intended to operate correctly in processor-ordered processors (such as the Pentium 4, Intel Xeon, and P6 
family processors) should not depend on the relatively strong ordering of the Pentium or Intel486 processors. 
Instead, it should ensure that accesses to shared variables that are intended to control concurrent execution 
among processors are explicitly required to obey program ordering through the use of appropriate locking or seri- 
alizing operations (see Section 8.2.5, “Strengthening or Weakening the Memory-Ordering Model”). 


8.2.2 Memory Ordering in P6 and More Recent Processor Families 


The Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium 4, and P6 family processors also use a processor-ordered 
memory-ordering model that can be further defined as “write ordered with store-buffer forwarding.” This model can 
be characterized as follows. 


In a single-processor system for memory regions defined as write-back cacheable, the memory-ordering model 
respects the following principles (Note the memory-ordering principles for single-processor and multiple- 
processor systems are written from the perspective of software executing on the processor, where the term 
“processor” refers to a logical processor. For example, a physical processor supporting multiple cores and/or Intel 
Hyper-Threading Technology is treated as a multi-processor systems. ): 


® Reads are not reordered with other reads. 
¢ Writes are not reordered with older reads. 
¢ Writes to memory are not reordered with other writes, with the following exceptions: 


— streaming stores (writes) executed with the non-temporal move instructions (MOVNTI, MOVNTQ, 
MOVNTDQ, MOVNTPS, and MOVNTPD); and 


— string operations (see Section 8.2.4.1). 


¢ No write to memory may be reordered with an execution of the CLFLUSH instruction; a write may be reordered 
with an execution of the CLFLUSHOPT instruction that flushes a cache line other than the one being written. 
Executions of the CLFLUSH instruction are not reordered with each other. Executions of CLFLUSHOPT that 
access different cache lines may be reordered with each other. An execution of CLFLUSHOPT may be reordered 
with an execution of CLFLUSH that accesses a different cache line. 


® Reads may be reordered with older writes to different locations but not with older writes to the same location. 
® Reads or writes cannot be reordered with I/O instructions, locked instructions, or serializing instructions. 
¢ Reads cannot pass earlier LFENCE and MFENCE instructions. 


¢ Writes and executions of CLFLUSH and CLFLUSHOPT cannot pass earlier LFENCE, SFENCE, and MFENCE 
instructions. 


e LFENCE instructions cannot pass earlier reads. 
¢ SFENCE instructions cannot pass earlier writes or executions of CLFLUSH and CLFLUSHOPT. 
¢ MFENCE instructions cannot pass earlier reads, writes, or executions of CLFLUSH and CLFLUSHOPT. 


1. Earlier versions of this manual specified that writes to memory may be reordered with executions of the CLFLUSH instruction. No 
processors implementing the CLFLUSH instruction allow such reordering. 
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In a multiple-processor system, the following ordering principles apply: 

¢ Individual processors use the same ordering principles as in a single-processor system. 

¢ Writes by a single processor are observed in the same order by all processors. 

¢ Writes from an individual processor are NOT ordered with respect to the writes from other processors. 
¢ Memory ordering obeys causality (memory ordering respects transitive visibility). 

¢ Any two stores are seen in a consistent order by processors other than those performing the stores 

¢ Locked instructions have a total order. 


See the example in Figure 8-1. Consider three processors in a system and each processor performs three writes, 
one to each of three defined locations (A, B, and C). Individually, the processors perform the writes in the same 
program order, but because of bus arbitration and other memory access mechanisms, the order that the three 
processors write the individual memory locations can differ each time the respective code sequences are executed 
on the processors. The final values in location A, B, and C would possibly vary on each execution of the write 
sequence. 


The processor-ordering model described in this section is virtually identical to that used by the Pentium and 
Intel486 processors. The only enhancements in the Pentium 4, Intel Xeon, and P6 family processors are: 


¢ Added support for speculative reads, while still adhering to the ordering principles above. 
°  Store-buffer forwarding, when a read passes a write to the same memory location. 


* Out of order store from long string store and string move operations (see Section 8.2.4, “Fast-String Operation 
and Out-of-Order Stores,” below). 


Order of Writes From Individual Processors 


Processor #1 Processor #2 Processor #3 
Each processor — Write A.1 Write A.2 Write A.3 
is guaranteed to ~~ | Write B14 Write B.2 Write B.3 
perform writes in _— Write C.1 Write C.2 Write C.3 


program order. 


Example of order of actual writes 
from all processors to memory 


Writes are in order Write A.1 — 

with respect to Write B.1 

individual processes. Write A.2 Writes from all 
Write A.3 processors are 
Write C.1. |-» not guaranteed 
Write B.2 to occur ina 
Write C.2 particular order. 
Write B.3 
Write C.3 — 


Figure 8-1. Example of Write Ordering in Multiple-Processor Systems 


NOTE 


In P6 processor family, store-buffer forwarding to reads of WC memory from streaming stores to the same address 
does not occur due to errata. 


8.2.3 Examples Illustrating the Memory-Ordering Principles 


This section provides a set of examples that illustrate the behavior of the memory-ordering principles introduced in 
Section 8.2.2. They are designed to give software writers an understanding of how memory ordering may affect 
the results of different sequences of instructions. 
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These examples are limited to accesses to memory regions defined as write-back cacheable (WB). (Section 8.2.3.1 
describes other limitations on the generality of the examples.) The reader should understand that they describe 
only software-visible behavior. A logical processor may reorder two accesses even if one of examples indicates that 
they may not be reordered. Such an example states only that software cannot detect that such a reordering 
occurred. Similarly, a logical processor may execute a memory access more than once as long as the behavior 
visible to software is consistent with a single execution of the memory access. 


8.2.3.1 Assumptions, Terminology, and Notation 


As noted above, the examples in this section are limited to accesses to memory regions defined as write-back 
cacheable (WB). They apply only to ordinary loads stores and to locked read-modify-write instructions. They do not 
necessarily apply to any of the following: out-of-order stores for string instructions (see Section 8.2.4); accesses 
with a non-temporal hint; reads from memory by the processor as part of address translation (e.g., page walks); 
and updates to segmentation and paging structures by the processor (e.g., to update “accessed” bits). 


The principles underlying the examples in this section apply to individual memory accesses and to locked read- 
modify-write instructions. The Intel-64 memory-ordering model guarantees that, for each of the following 
memory-access instructions, the constituent memory operation appears to execute as a single memory access: 


¢ Instructions that read or write a single byte. 

¢ Instructions that read or write a word (2 bytes) whose address is aligned on a 2 byte boundary. 

¢ Instructions that read or write a doubleword (4 bytes) whose address is aligned on a 4 byte boundary. 
¢ Instructions that read or write a quadword (8 bytes) whose address is aligned on an 8 byte boundary. 


Any locked instruction (either the XCHG instruction or another read-modify-write instruction with a LOCK prefix) 
appears to execute as an indivisible and uninterruptible sequence of load(s) followed by store(s) regardless of 
alignment. 


Other instructions may be implemented with multiple memory accesses. From a memory-ordering point of view, 
there are no guarantees regarding the relative order in which the constituent memory accesses are made. There is 
also no guarantee that the constituent operations of a store are executed in the same order as the constituent 
operations of a load. 


Section 8.2.3.2 through Section 8.2.3.7 give examples using the MOV instruction. The principles that underlie 
these examples apply to load and store accesses in general and to other instructions that load from or store to 
memory. Section 8.2.3.8 and Section 8.2.3.9 give examples using the XCHG instruction. The principles that 
underlie these examples apply to other locked read-modify-write instructions. 


This section uses the term “processor” is to refer to a logical processor. The examples are written using Intel-64 
assembly-language syntax and use the following notational conventions: 


¢ Arguments beginning with an “r”, such as ri or r2 refer to registers (e.g., EAX) visible only to the processor 
being considered. 


* Memory locations are denoted with x, y, z. 
¢ Stores are written as mov [ _ x], val, which implies that va/ is being stored into the memory location x. 


° Loads are written as mov r, [ __x], which implies that the contents of the memory location x are being loaded 
into the register r. 


As noted earlier, the examples refer only to software visible behavior. When the succeeding sections make state- 
ment such as “the two stores are reordered,” the implication is only that “the two stores appear to be reordered 
from the point of view of software.” 
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8.2.3.2 Neither Loads Nor Stores Are Reordered with Like Operations 


The Intel-64 memory-ordering model allows neither loads nor stores to be reordered with the same kind of opera- 
tion. That is, it ensures that loads are seen in program order and that stores are seen in program order. This is illus- 
trated by the following example: 


Example 8-1. Stores Are Not Reordered with Other Stores 


Processor 0 Processor 1 
mov [_x], 1 mov r1,[_y] 
mov [_y], 1 mov r2, [_x] 
Initially x = y = 0 


r1=1 andr2 = Ois not allowed 


The disallowed return values could be exhibited only if processor 0's two stores are reordered (with the two loads 
occurring between them) or if processor 1’s two loads are reordered (with the two stores occurring between them). 


If ri = 1, the store to y occurs before the load from y. Because the Intel-64 memory-ordering model does not allow 
stores to be reordered, the earlier store to x occurs before the load from y. Because the Intel-64 memory-ordering 
model does not allow loads to be reordered, the store to x also occurs before the later load from x. This r2 = 1. 


8.2.3.3 Stores Are Not Reordered With Earlier Loads 


The Intel-64 memory-ordering model ensures that a store by a processor may not occur before a previous load by 
the same processor. This is illustrated by the following example: 


Example 8-2. Stores Are Not Reordered with Older Loads 


Processor O Processor 1 
mov r1, [ __x] mov r2, [_y] 
mov [_y], 1 mov [ _x], 1 
Initially x = y =O 


r1 = 1 andre = 1 is not allowed 


Assume rl = 1. 
° Because ri = 1, processor 1’s store to x occurs before processor 0’s load from x. 


° Because the Intel-64 memory-ordering model prevents each store from being reordered with the earlier load 
by the same processor, processor 1's load from y occurs before its store to x. 


¢ Similarly, processor 0’s load from x occurs before its store to y. 
¢ Thus, processor 1’s load from y occurs before processor 0's store to y, implying r2 = 0. 


8.2.3.4 Loads May Be Reordered with Earlier Stores to Different Locations 


The Intel-64 memory-ordering model allows a load to be reordered with an earlier store to a different location. 
However, loads are not reordered with stores to the same location. 


The fact that a load may be reordered with an earlier store to a different location is illustrated by the following 
example: 


Example 8-3. Loads May be Reordered with Older Stores 


1 =O andr2 = Ois allowed 


Processor 0 Processor 1 
mov [ _x], 1 mov [ _y], 1 
movri,[_y] mov r2, [ _x] 
Initially x = y =O 
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At each processor, the load and the store are to different locations and hence may be reordered. Any interleaving 


of the operations is thus allowed. One such interleaving has the two loads occurring before the two stores. This 
would result in each load returning value 0. 


The fact that a load may not be reordered with an earlier store to the same location is illustrated by the following 
example: 


Example 8-4. Loads Are not Reordered with Older Stores to the Same Location 
Processor 0 


mov [ _x], 1 
mov r1,[_x] 


Initially x = 0 
r1 = Ois not allowed 


The Intel-64 memory-ordering model does not allow the load to be reordered with the earlier store because the 
accesses are to the same location. Therefore, rl = 1 must hold. 


8.2.3.5 Intra-Processor Forwarding Is Allowed 


The memory-ordering model allows concurrent stores by two processors to be seen in different orders by those two 


processors; specifically, each processor may perceive its own store occurring before that of the other. This is illus- 
trated by the following example: 


Example 8-5. Intra-Processor Forwarding is Allowed 


Processor 0 Processor 1 
mov [ _x], 1 mov [ _y], 1 
mov r1, [ __x] mov r3, [_y] 
mov r2,[_y] mov r4, [ __x] 
Initially x = y =0 


r2 = O and r4 = Ois allowed 


The memory-ordering model imposes no constraints on the order in which the two stores appear to execute by the 
two processors. This fact allows processor 0 to see its store before seeing processor 1's, while processor 1 sees its 
store before seeing processor 0's. (Each processor is self consistent.) This allows r2 = 0 and r4 = 0. 


In practice, the reordering in this example can arise as a result of store-buffer forwarding. While a store is tempo- 


rarily held in a processor's store buffer, it can satisfy the processor's own loads but is not visible to (and cannot 
satisfy) loads by other processors. 


8.2.3.6 Stores Are Transitively Visible 


The memory-ordering model ensures transitive visibility of stores; stores that are causally related appear to all 
processors to occur in an order consistent with the causality relation. This is illustrated by the following example: 
Example 8-6. Stores Are Transitively Visible 


Processor 0 Processor 1 Processor 2 
mov [ _x], 1 mov r1, [_x] 
mov [_y], 1 mov r2, [ _y] 
mov r3, [_x] 
Initially x = y =0 
r1=1,r2 = 1,73 = Ois not allowed 


Assume that ri = 1 and r2 = 1. 


Because ri = 1, processor 0’s store occurs before processor 1’s load. 
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¢ Because the memory-ordering model prevents a store from being reordered with an earlier load (see Section 
8.2.3.3), processor 1’s load occurs before its store. Thus, processor 0’s store causally precedes processor 1's 
store. 


¢ Because processor 0’s store causally precedes processor 1’s store, the memory-ordering model ensures that 
processor 0’s store appears to occur before processor 1’s store from the point of view of all processors. 


¢ Because r2 = 1, processor 1’s store occurs before processor 2's load. 


° Because the Intel-64 memory-ordering model prevents loads from being reordered (see Section 8.2.3.2), 
processor 2’s load occur in order. 


¢ The above items imply that processor 0’s store to x occurs before processor 2's load from x. This implies that 
r3 = 1. 


8.2.3.7 Stores Are Seen in a Consistent Order by Other Processors 


As noted in Section 8.2.3.5, the memory-ordering model allows stores by two processors to be seen in different 
orders by those two processors. However, any two stores must appear to execute in the same order to all proces- 
sors other than those performing the stores. This is illustrated by the following example: 


Example 8-7. Stores Are Seen in a Consistent Order by Other Processors 


Processor 0 


Processor 1 


Processor 2 


Processor 3 


mov [ _x], 1 


mov [ _y], 1 


mov r1, [ __x] 
mov r2, [_y] 


mov r3, [_y] 
mov r4, [_x] 


Initially x = y =O 


rl =1,r2 =0,r3 = 1, r4 = Ois not allowed 


By the principles discussed in Section 8.2.3.2, 


® processor 2's first and second load cannot be reordered, 


¢ processor 3’s first and second load cannot be reordered. 
¢ Ifri=1andr2 =0, processor 0’s store appears to precede processor 1’s store with respect to processor 2. 
¢ Similarly, r3 = 1 and r4 = 0 imply that processor 1's store appears to precede processor 0’s store with respect 


to processor 1. 


Because the memory-ordering model ensures that any two stores appear to execute in the same order to all 
processors (other than those performing the stores), this set of return values is not allowed 


8.2.3.8 Locked Instructions Have a Total Order 


The memory-ordering model ensures that all processors agree on a single execution order of all locked instruc- 
tions, including those that are larger than 8 bytes or are not naturally aligned. This is illustrated by the following 


example: 


Example 8-8. Locked Instructions Have a Total Order 


Processor 0 


Processor 1 


Processor 2 


Processor 3 


xchg [ _x], 1 


xchg [ _y], r2 


mov r3, [ _x] 
mov r4,[_y] 


mov r5, [_y] 
mov r6, [_x] 


Initially r1 =r2 =1,x=y=0 


r3 = 1,r4=0,r5 = 1, r6 = Ois not allowed 


Processor 2 and processor 3 must agree on the order of the two executions of XCHG. Without loss of generality, 
suppose that processor 0’s XCHG occurs first. 


¢ If r5 = 1, processor 1’s XCHG into y occurs before processor 3’s load from y. 
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° Because the Intel-64 memory-ordering model prevents loads from being reordered (see Section 8.2.3.2), 
processor 3’s loads occur in order and, therefore, processor 1’s XCHG occurs before processor 3’s load from x. 


¢® Since processor 0’s XCHG into x occurs before processor 1’s XCHG (by assumption), it occurs before 
processor 3’s load from x. Thus, r6 = 1. 


A similar argument (referring instead to processor 2’s loads) applies if processor 1’s XCHG occurs before 
processor 0’s XCHG. 


8.2.3.9 Loads and Stores Are Not Reordered with Locked Instructions 


The memory-ordering model prevents loads and stores from being reordered with locked instructions that execute 
earlier or later. The examples in this section illustrate only cases in which a locked instruction is executed before a 
load or a store. The reader should note that reordering is prevented also if the locked instruction is executed after 
a load or a store. 


The first example illustrates that loads may not be reordered with earlier locked instructions: 
Example 8-9. Loads Are not Reordered with Locks 


Processor 0 


Processor 1 


xchg [ _x], 71 
mov r2, [_y] 


xchg [ _y], r3 
mov r4, [ __x] 


Initially x = y=0,r1 =r3=1 
r2 = 0 and r4 = Ois not allowed 


As explained in Section 8.2.3.8, there is a total order of the executions of locked instructions. Without loss of 
generality, suppose that processor 0’s XCHG occurs first. 


Because the Intel-64 memory-ordering model prevents processor 1’s load from being reordered with its earlier 
XCHG, processor 0’s XCHG occurs before processor 1's load. This implies r4 = 1. 


A similar argument (referring instead to processor 2's accesses) applies if processor 1’s XCHG occurs before 
processor 0’s XCHG. 


The second example illustrates that a store may not be reordered with an earlier locked instruction: 
Example 8-10. Stores Are not Reordered with Locks 


Processor 0 


Processor 1 


xchg [ _x], 71 
mov [_y], 1 


mov r2, [_y] 
mov r3, [ _x] 


Initially x = y=0,r1 =1 
r2 = 1 andr3 = Ois not allowed 


Assume r2 = 1. 
°® Because r2 = 1, processor 0’s store to y occurs before processor 1’s load from y. 


° Because the memory-ordering model prevents a store from being reordered with an earlier locked instruction, 
processor 0’s XCHG into x occurs before its store to y. Thus, processor 0’s XCHG into x occurs before 
processor 1's load from y. 


°® Because the memory-ordering model prevents loads from being reordered (see Section 8.2.3.2), processor 1's 
loads occur in order and, therefore, processor 1’s XCHG into x occurs before processor 1’s load from x. Thus, 
r3 = 1. 


8.2.4 Fast-String Operation and Out-of-Order Stores 


Section 7.3.9.3 of Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 described an optimi- 
zation of repeated string operations called fast-string operation. 
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As explained in that section, the stores produced by fast-string operation may appear to execute out of order. Soft- 
ware dependent upon sequential store ordering should not use string operations for the entire data structure to be 
stored. Data and semaphores should be separated. Order-dependent code should write to a discrete semaphore 
variable after any string operations to allow correctly ordered data to be seen by all processors. Atomicity of load 
and store operations is guaranteed only for native data elements of the string with native data size, and only if they 
are included in a single cache line. 


Section 8.2.4.1 and Section 8.2.4.2 provide further explain and examples. 


8.2.4.1 Memory-Ordering Model for String Operations on Write-Back (WB) Memory 


This section deals with the memory-ordering model for string operations on write-back (WB) memory for the Intel 
64 architecture. 


The memory-ordering model respects the follow principles: 
1. Stores within a single string operation may be executed out of order. 


2. Stores from separate string operations (for example, stores from consecutive string operations) do not execute 
out of order. All the stores from an earlier string operation will complete before any store from a later string 
operation. 


3. String operations are not reordered with other store operations. 


Fast string operations (e.g. string operations initiated with the MOVS/STOS instructions and the REP prefix) may be 
interrupted by exceptions or interrupts. The interrupts are precise but may be delayed - for example, the interrup- 
tions may be taken at cache line boundaries, after every few iterations of the loop, or after operating on every few 
bytes. Different implementations may choose different options, or may even choose not to delay interrupt 
handling, so software should not rely on the delay. When the interrupt/trap handler is reached, the source/destina- 
tion registers point to the next string element to be operated on, while the EIP stored in the stack points to the 
string instruction, and the ECX register has the value it held following the last successful iteration. The return from 
that trap/interrupt handler should cause the string instruction to be resumed from the point where it was inter- 
rupted. 


The string operation memory-ordering principles, (item 2 and 3 above) should be interpreted by taking the incor- 
ruptibility of fast string operations into account. For example, if a fast string operation gets interrupted after k iter- 
ations, then stores performed by the interrupt handler will become visible after the fast string stores from iteration 
0 to k, and before the fast string stores from the (k+1)th iteration onward. 


Stores within a single string operation may execute out of order (item 1 above) only if fast string operation is 
enabled. Fast string operations are enabled/disabled through the IA32_MISC_ENABLE model specific register. 


8.2.4.2 Examples Illustrating Memory-Ordering Principles for String Operations 
The following examples uses the same notation and convention as described in Section 8.2.3.1. 


In Example 8-11, processor 0 does one round of (128 iterations) doubleword string store operation via rep:stosd, 
writing the value 1 (value in EAX) into a block of 512 bytes from location _x (kept in ES:EDI) in ascending order. 
Since each operation stores a doubleword (4 bytes), the operation is repeated 128 times (value in ECX). The block 
of memory initially contained 0. Processor 1 is reading two memory locations that are part of the memory block 
being updated by processor 0, i.e, reading locations in the range _x to (_x+511). 


Example 8-11. Stores Within a String Operation May be Reordered 
Processor 0 Processor 1 


rep:stosd [ _x] mov rl, [_z] 


mov r2,[_y] 


Initially on processor 0: EAX = 1, ECX=128, ES:EDI =_x 
Initially [_x] to 511[_x]= 0, __x <=_y <_z<_x+512 
r1 = 1 andr2 = Ois allowed 


Vol. 3A 8-13 


MULTIPLE-PROCESSOR MANAGEMENT 


It is possible for processor 1 to perceive that the repeated string stores in processor 0 are happening out of order. 
Assume that fast string operations are enabled on processor 0. 


In Example 8-12, processor 0 does two separate rounds of rep stosd operation of 128 doubleword stores, writing 
the value 1 (value in EAX) into the first block of 512 bytes from location _x (kept in ES:EDI) in ascending order. It 
then writes 1 into a second block of memory from (_x+512) to (_x+1023). All of the memory locations initially 
contain 0. The block of memory initially contained 0. Processor 1 performs two load operations from the two blocks 
of memory. 


Example 8-12. Stores Across String Operations Are not Reordered 


Processor 0 Processor 1 
rep:stosd [ _x] 
mov rl, [_z] 
mov ecx, $128 
mov r2, [_y] 
rep:stosd 512[ _x] 


Initially on processor O: EAX = 1, ECX=128, ES:EDI =_x 
Initially [_x] to 1023[_x]= 0, _x <=_y < _xt+512 <_z<_x+1024 
r1 = 1 andr2 = Ois not allowed 


It is not possible in the above example for processor 1 to perceive any of the stores from the later string operation 
(to the second 512 block) in processor 0 before seeing the stores from the earlier string operation to the first 512 
block. 


The above example assumes that writes to the second block (_x+512 to _x+1023) does not get executed while 
processor 0’s string operation to the first block has been interrupted. If the string operation to the first block by 
processor 0 is interrupted, and a write to the second memory block is executed by the interrupt handler, then that 
change in the second memory block will be visible before the string operation to the first memory block resumes. 


In Example 8-13, processor 0 does one round of (128 iterations) doubleword string store operation via rep:stosd, 
writing the value 1 (value in EAX) into a block of 512 bytes from location _x (kept in ES:EDI) in ascending order. It 
then writes to a second memory location outside the memory block of the previous string operation. Processor 1 
performs two read operations, the first read is from an address outside the 512-byte block but to be updated by 
processor 0, the second ready is from inside the block of memory of string operation. 


Example 8-13. String Operations Are not Reordered with later Stores 


Processor 0 Processor 1 


rep:stosd [ _x] mov r1, [_z] 


mov [_z], $1 mov r2, [_y] 


Initially on processor O: EAX = 1, ECX=128, ES:EDI =_x 
Initially [_y] = [_z] = 0, [_x] to 511[_x]= 0, _x <=_y < _x+512, _z is a separate memory location 


r1 = 1 andr2 = Ois not allowed 


Processor 1 cannot perceive the later store by processor 0 until it sees all the stores from the string operation. 
Example 8-13 assumes that processor 0’s store to [_z] is not executed while the string operation has been inter- 
rupted. If the string operation is interrupted and the store to [_z] by processor 0 is executed by the interrupt 
handler, then changes to [_z] will become visible before the string operation resumes. 


Example 8-14 illustrates the visibility principle when a string operation is interrupted. 
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Example 8-14. Interrupted String Operation 


Processor 0 Processor 1 
rep:stosd [ _ x] // interrupted before es:edi reach _y mov r, [_z] 
mov [_z], $1 // interrupt handler mov r2,[_y] 


Initially on processor 0: EAX = 1, ECX=128, ES:EDI =_x 
Initially [_y] = [_z] = 0, [_x] to 511[_x]= 0, _x <= _y < __x+512,_2 is a separate memory location 
r1 = 1 andr2 = Ois allowed 


In Example 8-14, processor 0 started a string operation to write to a memory block of 512 bytes starting at address 
_x. Processor 0 got interrupted after k iterations of store operations. The address _y has not yet been updated by 
processor 0 when processor 0 got interrupted. The interrupt handler that took control on processor 0 writes to the 
address _z. Processor 1 may see the store to _z from the interrupt handler, before seeing the remaining stores to 
the 512-byte memory block that are executed when the string operation resumes. 


Example 8-15 illustrates the ordering of string operations with earlier stores. No store from a string operation can 
be visible before all prior stores are visible. 


Example 8-15. String Operations Are not Reordered with Earlier Stores 
Processor 0 Processor 1 


mov [_z], $1 mov 1, [_y] 

rep:stosd [ _x] mov r2, [_z] 

Initially on processor 0: EAX = 1, ECX=128, ES:EDI =_x 

Initially [_y] = [_z] = 0, [_x] to 511[_x]= 0, __x <= _y < __x+512,_2 is a separate memory location 
r1 = 1 andr2 = Ois not allowed 


8.2.5 Strengthening or Weakening the Memory-Ordering Model 


The Intel 64 and JA-32 architectures provide several mechanisms for strengthening or weakening the memory- 
ordering model to handle special programming situations. These mechanisms include: 


¢ The I/O instructions, locking instructions, the LOCK prefix, and serializing instructions force stronger ordering 
on the processor. 


¢ The SFENCE instruction (introduced to the IA-32 architecture in the Pentium III processor) and the LFENCE and 
MFENCE instructions (introduced in the Pentium 4 processor) provide memory-ordering and serialization 
capabilities for specific types of memory operations. 


° The memory type range registers (MTRRs) can be used to strengthen or weaken memory ordering for specific 
area of physical memory (see Section 11.11, “Memory Type Range Registers (MTRRs)”). MTRRs are available 
only in the Pentium 4, Intel Xeon, and P6 family processors. 


° The page attribute table (PAT) can be used to strengthen memory ordering for a specific page or group of pages 
(see Section 11.12, “Page Attribute Table (PAT)”). The PAT is available only in the Pentium 4, Intel Xeon, and 
Pentium III processors. 


These mechanisms can be used as follows: 


Memory mapped devices and other I/O devices on the bus are often sensitive to the order of writes to their I/O 
buffers. I/O instructions can be used to (the IN and OUT instructions) impose strong write ordering on such 
accesses as follows. Prior to executing an I/O instruction, the processor waits for all previous instructions in the 
program to complete and for all buffered writes to drain to memory. Only instruction fetch and page tables walks 
can pass I/O instructions. Execution of subsequent instructions do not begin until the processor determines that 
the I/O instruction has been completed. 
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Synchronization mechanisms in multiple-processor systems may depend upon a strong memory-ordering model. 
Here, a program can use a locking instruction such as the XCHG instruction or the LOCK prefix to ensure that a 
read-modify-write operation on memory is carried out atomically. Locking operations typically operate like I/O 
operations in that they wait for all previous instructions to complete and for all buffered writes to drain to memory 
(see Section 8.1.2, “Bus Locking”). 


Program synchronization can also be carried out with serializing instructions (see Section 8.3). These instructions 
are typically used at critical procedure or task boundaries to force completion of all previous instructions before a 
jump to a new section of code or a context switch occurs. Like the I/O and locking instructions, the processor waits 
until all previous instructions have been completed and all buffered writes have been drained to memory before 
executing the serializing instruction. 


The SFENCE, LFENCE, and MFENCE instructions provide a performance-efficient way of ensuring load and store 
memory ordering between routines that produce weakly-ordered results and routines that consume that data. The 
functions of these instructions are as follows: 


° SFENCE — Serializes all store (write) operations that occurred prior to the SFENCE instruction in the program 
instruction stream, but does not affect load operations. 


¢ LFENCE — Serializes all load (read) operations that occurred prior to the LFENCE instruction in the program 
instruction stream, but does not affect store operations. 


¢ MFENCE — Serializes all store and load operations that occurred prior to the MFENCE instruction in the 
program instruction stream. 


Note that the SFENCE, LFENCE, and MFENCE instructions provide a more efficient method of controlling memory 
ordering than the CPUID instruction. 


The MTRRs were introduced in the P6 family processors to define the cache characteristics for specified areas of 
physical memory. The following are two examples of how memory types set up with MTRRs can be used strengthen 
or weaken memory ordering for the Pentium 4, Intel Xeon, and P6 family processors: 


¢ The strong uncached (UC) memory type forces a strong-ordering model on memory accesses. Here, all reads 
and writes to the UC memory region appear on the bus and out-of-order or speculative accesses are not 
performed. This memory type can be applied to an address range dedicated to memory mapped I/O devices to 
force strong memory ordering. 


¢ For areas of memory where weak ordering is acceptable, the write back (WB) memory type can be chosen. 
Here, reads can be performed speculatively and writes can be buffered and combined. For this type of memory, 
cache locking is performed on atomic (locked) operations that do not split across cache lines, which helps to 
reduce the performance penalty associated with the use of the typical synchronization instructions, such as 
XCHG, that lock the bus during the entire read-modify-write operation. With the WB memory type, the XCHG 
instruction locks the cache instead of the bus if the memory access is contained within a cache line. 


The PAT was introduced in the Pentium III processor to enhance the caching characteristics that can be assigned to 
pages or groups of pages. The PAT mechanism typically used to strengthen caching characteristics at the page level 
with respect to the caching characteristics established by the MTRRs. Table 11-7 shows the interaction of the PAT 
with the MTRRs. 


Intel recommends that software written to run on Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium 4, Intel 
Xeon, and P6 family processors assume the processor-ordering model or a weaker memory-ordering model. The 
Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium 4, Intel Xeon, and P6 family processors do not implement a 
strong memory-ordering model, except when using the UC memory type. Despite the fact that Pentium 4, Intel 
Xeon, and P6 family processors support processor ordering, Intel does not guarantee that future processors will 
support this model. To make software portable to future processors, it is recommended that operating systems 
provide critical region and resource control constructs and API’s (application program interfaces) based on I/O, 
locking, and/or serializing instructions be used to synchronize access to shared areas of memory in multiple- 
processor systems. Also, software should not depend on processor ordering in situations where the system hard- 
ware does not support this memory-ordering model. 


2. Specifically, LFENCE does not execute until all prior instructions have completed locally, and no later instruction begins execution 
until LFENCE completes. As a result, an instruction that loads from memory and that precedes an LFENCE receives data from mem- 
ory prior to completion of the LFENCE. An LFENCE that follows an instruction that stores to memory might complete before the data 
being stored have become globally visible. Instructions following an LFENCE may be fetched from memory before the LFENCE, but 
they will not execute until the LFENCE completes. 
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8.3 SERIALIZING INSTRUCTIONS 


The Intel 64 and JA-32 architectures define several serializing instructions. These instructions force the 
processor to complete all modifications to flags, registers, and memory by previous instructions and to drain all 
buffered writes to memory before the next instruction is fetched and executed. For example, when a MOV to 
control register instruction is used to load a new value into control register CRO to enable protected mode, the 
processor must perform a serializing operation before it enters protected mode. This serializing operation ensures 
that all operations that were started while the processor was in real-address mode are completed before the switch 
to protected mode is made. 


The concept of serializing instructions was introduced into the IA-32 architecture with the Pentium processor to 
support parallel instruction execution. Serializing instructions have no meaning for the Intel486 and earlier proces- 
sors that do not implement parallel instruction execution. 


It is important to note that executing of serializing instructions on P6 and more recent processor families constrain 
speculative execution because the results of speculatively executed instructions are discarded. The following 
instructions are serializing instructions: 


° Privileged serializing instructions — INVD, INVEPT, INVLPG, INVVPID, LGDT, LIDT, LLDT, LTR, MOV (to 
control register, with the exception of MOV CR8?), MOV (to debug register), WBINVD, and WRMSR4. 


° Non-privileged serializing instructions — CPUID, IRET, and RSM. 


When the processor serializes instruction execution, it ensures that all pending memory transactions are 
completed (including writes stored in its store buffer) before it executes the next _instruction. Nothing can pass a 
serializing instruction and a serializing instruction cannot pass any other instruction (read, write, instruction fetch, 
or I/O). For example, CPUID can be executed at any privilege level to serialize instruction execution with no effect 
on program flow, except that the EAX, EBX, ECX, and EDX registers are modified. 


The following instructions are memory-ordering instructions, not serializing instructions. These drain the data 
memory subsystem. They do not serialize the instruction execution stream:> 


¢ Non-privileged memory-ordering instructions — SFENCE, LFENCE, and MFENCE. 


The SFENCE, LFENCE, and MFENCE instructions provide more granularity in controlling the serialization of memory 
loads and stores (see Section 8.2.5, “Strengthening or Weakening the Memory-Ordering Model”). 


The following additional information is worth noting regarding serializing instructions: 


¢ The processor does not write back the contents of modified data in its data cache to external memory when it 
serializes instruction execution. Software can force modified data to be written back by executing the WBINVD 
instruction, which is a serializing instruction. The amount of time or cycles for WBINVD to complete will vary 
due to the size of different cache hierarchies and other factors. As a consequence, the use of the WBINVD 
instruction can have an impact on interrupt/event response time. 


¢ When an instruction is executed that enables or disables paging (that is, changes the PG flag in control register 
CRO), the instruction should be followed by a jump instruction. The target instruction of the jump instruction is 
fetched with the new setting of the PG flag (that is, paging is enabled or disabled), but the jump instruction 
itself is fetched with the previous setting. The Pentium 4, Intel Xeon, and P6 family processors do not require 
the jump operation following the move to register CRO (because any use of the MOV instruction in a Pentium 4, 
Intel Xeon, or P6 family processor to write to CRO is completely serializing). However, to maintain backwards 
and forward compatibility with code written to run on other IA-32 processors, it is recommended that the jump 
operation be performed. 


3. MOV CR8 is not defined architecturally as a serializing instruction. 

4. Anexecution of WRMSR to any non-serializing MSR is not serializing. Non-serializing MSRs include the following: [A32_SPEC_CTRL 
MSR (MSR index 48H), [A32_PRED_CMD MSR (MSR index 49H), |A32_TSX_CTRL MSR (MSR index 122H), [A32_TSC_DEADLINE MSR 
(MSR index 6E0H), IA32_PKRS MSR (MSR index 6€1H), [A32_HwWP_REQUEST MSR (MSR index 774H), or any of the x2APIC MSRs 
(MSR indices 802H to 83FH). 

5. ULFENCE does provide some guarantees on instruction ordering. It does not execute until all prior instructions have completed locally, 
and no later instruction begins execution until LFENCE completes. 
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¢ Whenever an instruction is executed to change the contents of CR3 while paging is enabled, the next instruction 
is fetched using the translation tables that correspond to the new value of CR3. Therefore the next instruction 
and the sequentially following instructions should have a mapping based upon the new value of CR3. (Global 
entries in the TLBs are not invalidated, see Section 4.10.4, “Invalidation of TLBs and Paging-Structure 
Caches.”) 


¢ The Pentium processor and more recent processor families use branch-prediction techniques to improve 
performance by prefetching the destination of a branch instruction before the branch instruction is executed. 
Consequently, instruction execution is not deterministically serialized when a branch instruction is executed. 


8.4 MULTIPLE-PROCESSOR (MP) INITIALIZATION 


The IA-32 architecture (beginning with the P6 family processors) defines a multiple-processor (MP) initialization 
protocol called the Multiprocessor Specification Version 1.4. This specification defines the boot protocol to be used 
by IA-32 processors in multiple-processor systems. (Here, multiple processors is defined as two or more proces- 
sors.) The MP initialization protocol has the following important features: 


¢ It supports controlled booting of multiple processors without requiring dedicated system hardware. 


¢ It allows hardware to initiate the booting of a system without the need for a dedicated signal or a predefined 
boot processor. 


¢ It allows all IA-32 processors to be booted in the same manner, including those supporting Intel Hyper- 
Threading Technology. 


¢ The MP initialization protocol also applies to MP systems using Intel 64 processors. 


The mechanism for carrying out the MP initialization protocol differs depending on the Intel processor generations. 
The following bullets summarizes the evolution of the changes: 


° For P6 family or older processors supporting MP operations— The selection of the BSP and APs (see 
Section 8.4.1, “BSP and AP Processors”) is handled through arbitration on the APIC bus, using BIPI and FIPI 
messages. These processor generations have CPUID signatures of (family=06H, extended_model=0, 
model<=ODH), or family <O6H. See Section 8.11.1, “Overview of the MP Initialization Process For P6 Family 
Processors” for a complete discussion of MP initialization for P6 family processors. 


¢ Early generations of IA processors with family OFH — The selection of the BSP and APs (see Section 
8.4.1, “BSP and AP Processors”) is handled through arbitration on the system bus, using BIPI and FIPI 
messages (see Section 8.4.3, “MP Initialization Protocol Algorithm for MP Systems”). These processor 
generations have CPUID signatures of family=OFH, model=0H, stepping<=09H. 


¢ Later generations of IA processors with family OFH, and IA processors with system bus — The 
selection of the BSP and APs is handled through a special system bus cycle, without using BIPI and FIPI 
message arbitration (see Section 8.4.3, “MP Initialization Protocol Algorithm for MP Systems”). These 
processor generations have CPUID signatures of family=OFH with (model=0H, stepping>=O0AH) or (model >0, 
all steppings); or family=O6H, extended_model=0, model>=OEH. 


¢ All other modern IA processor generations supporting MP operations— The selection of the BSP and 
APs in the system is handled by platform-specific arrangement of the combination of hardware, BIOS, and/or 
configuration input options. The basis of the selection mechanism is similar to those of the Later generations of 
family OFH and other Intel processor using system bus (see Section 8.4.3, “MP Initialization Protocol Algorithm 
for MP Systems”). These processor generations have CPUID signatures of family=O6H, extended_model>0. 


The family, model, and stepping ID for a processor is given in the EAX register when the CPUID instruction is 
executed with a value of 1 in the EAX register. 


8.4.1 BSP and AP Processors 


The MP initialization protocol defines two classes of processors: the bootstrap processor (BSP) and the application 
processors (APs). Following a power-up or RESET of an MP system, system hardware dynamically selects one of the 
processors on the system bus as the BSP. The remaining processors are designated as APs. 


As part of the BSP selection mechanism, the BSP flag is set in the IA32_APIC_BASE MSR (see Figure 10-5) of the 
BSP, indicating that it is the BSP. This flag is cleared for all other processors. 
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The BSP executes the BIOS’s boot-strap code to configure the APIC environment, sets up system-wide data struc- 
tures, and starts and initializes the APs. When the BSP and APs are initialized, the BSP then begins executing the 
operating-system initialization code. 


Following a power-up or reset, the APs complete a minimal self-configuration, then wait for a startup signal (a SIPI 
message) from the BSP processor. Upon receiving a SIPI message, an AP executes the BIOS AP configuration code, 
which ends with the AP being placed in halt state. 


For Intel 64 and IA-32 processors supporting Intel Hyper-Threading Technology, the MP initialization protocol treats 
each of the logical processors on the system bus or coherent link domain as a separate processor (with a unique 
APIC ID). During boot-up, one of the logical processors is selected as the BSP and the remainder of the logical 
processors are designated as APs. 


8.4.2 MP Initialization Protocol Requirements and Restrictions 


The MP initialization protocol imposes the following requirements and restrictions on the system: 


¢ The MP protocol is executed only after a power-up or RESET. If the MP protocol has completed and a BSP is 
chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP protocol to be 
repeated. Instead, each logical processor examines its BSP flag (in the IA32_APIC_BASE MSR) to determine 
whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an 
AP). 


e All devices in the system that are capable of delivering interrupts to the processors must be inhibited from 
doing so for the duration of the MP initialization protocol. The time during which interrupts must be inhibited 
includes the window between when the BSP issues an INIT-SIPI-SIPI sequence to an AP and when the AP 
responds to the last SIPI in the sequence. 


8.4.3 MP Initialization Protocol Algorithm for MP Systems 


Following a power-up or RESET of an MP system, the processors in the system execute the MP initialization protocol 
algorithm to initialize each of the logical processors on the system bus or coherent link domain. In the course of 
executing this algorithm, the following boot-up and initialization operations are carried out: 


1. Each logical processor is assigned a unique APIC ID, based on system topology. The unique ID is a 32-bit value 
if the processor supports CPUID leaf OBH, otherwise the unique ID is an 8-bit value. (see Section 8.4.5, “Identi- 
fying Logical Processors in an MP System”). 


Each logical processor is assigned a unique arbitration priority based on its APIC ID. 


Each logical processor executes its internal BIST simultaneously with the other logical processors in the 
system. 


4. Upon completion of the BIST, the logical processors use a hardware-defined selection mechanism to select the 
BSP and the APs from the available logical processors on the system bus. The BSP selection mechanism differs 
depending on the family, model, and stepping IDs of the processors, as follows: 


— Later generations of IA processors within family OFH (see Section 8.4), IA processors with system bus 
(family=06H, extended_model=0, model>=OEH), or all other modern Intel processors (family=O6H, 
extended_model>0): 


¢ The logical processors begin monitoring the BNR# signal, which is toggling. When the BNR# pin stops 
toggling, each processor attempts to issue a NOP special cycle on the system bus. 


¢ The logical processor with the highest arbitration priority succeeds in issuing a NOP special cycle and is 
nominated the BSP. This processor sets the BSP flag in its IA32_APIC_BASE MSR, then fetches and 
begins executing BIOS boot-strap code, beginning at the reset vector (physical address FFFF FFFOH). 


¢ The remaining logical processors (that failed in issuing a NOP special cycle) are designated as APs. They 
leave their BSP flags in the clear state and enter a “wait-for-SIPI state.” 
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— Early generations of IA processors within family OFH (family=OFH, model=0H, stepping<=09H), P6 family 
or older processors supporting MP operations (family=06H, extended_model=0, model<=ODH; or family 
<06H): 


¢ Each processor broadcasts a BIPI to “all including self.” The first processor that broadcasts a BIPI (and 
thus receives its own BIPI vector), selects itself as the BSP and sets the BSP flag in its IA32_APIC_BASE 
MSR. (See Section 8.11.1, “Overview of the MP Initialization Process For P6 Family Processors” for a 
description of the BIPI, FIPI, and SIPI messages.) 


¢ The remainder of the processors (which were not selected as the BSP) are designated as APs. They 
leave their BSP flags in the clear state and enter a “wait-for-SIPI state.” 


* The newly established BSP broadcasts an FIPI message to “all including self,” which the BSP and APs 
treat as an end of MP initialization signal. Only the processor with its BSP flag set responds to the FIPI 
message. It responds by fetching and executing the BIOS boot-strap code, beginning at the reset vector 
(physical address FFFF FFFOH). 


5. As part of the boot-strap code, the BSP creates an ACPI table and/or an MP table and adds its initial APIC ID to 
these tables as appropriate. 


6. Atthe end of the boot-strap procedure, the BSP sets a processor counter to 1, then broadcasts a SIPI message 
to all the APs in the system. Here, the SIPI message contains a vector to the BIOS AP initialization code (at 
OOOVVOOOH, where VV is the vector contained in the SIPI message). 


7. The first action of the AP initialization code is to set up a race (among the APs) to a BIOS initialization 
semaphore. The first AP to the semaphore begins executing the initialization code. (See Section 8.4.4, “MP 
Initialization Example,” for semaphore implementation details.) As part of the AP initialization procedure, the 
AP adds its APIC ID number to the ACPI and/or MP tables as appropriate and increments the processor counter 
by 1. At the completion of the initialization procedure, the AP executes a CLI instruction and halts itself. 


8. When each of the APs has gained access to the semaphore and executed the AP initialization code, the BSP 
establishes a count for the number of processors connected to the system bus, completes executing the BIOS 
boot-strap code, and then begins executing operating-system boot-strap and start-up code. 


9. While the BSP is executing operating-system boot-strap and start-up code, the APs remain in the halted state. 
In this state they will respond only to INITs, NMIs, and SMIs. They will also respond to snoops and to assertions 
of the STPCLK# pin. 


The following section gives an example (with code) of the MP initialization protocol for of multiple processors oper- 
ating in an MP configuration. 


Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4 describes how to program the LINT[0:1] pins of the processor’s local APICs after an MP configu- 
ration has been completed. 


8.4.4 MP Initialization Example 


The following example illustrates the use of the MP initialization protocol used to initialize processors in an MP 
system after the BSP and APs have been established. The code runs on Intel 64 or IA-32 processors that use a 
protocol. This includes P6 Family processors, Pentium 4 processors, Intel Core Duo, Intel Core 2 Duo and Intel Xeon 
processors. 


The following constants and data definitions are used in the accompanying 
code examples. They are based on the addresses of the APIC registers defined in Table 10-1. 


ICR_LOW EQU OFEEO0300H 
SVR EQU OFEEOOOFOH 
APIC_ID EQU OFEEO0020H 
LVT3 EQU OFEE00370H 
APIC_LENABLED €QU0100H 
BOOT_ID DD? 

COUNT EQU OOH 

VACANT EQU OOH 
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8.4.4.1 Typical BSP Initialization Sequence 


After the BSP and APs have been selected (by means of a hardware protocol, see Section 8.4.3, “MP Initialization 
Protocol Algorithm for MP Systems”), the BSP begins executing BIOS boot-strap code (POST) at the normal IA-32 
architecture starting address (FFFF FFFOH). The boot-strap code typically performs the following operations: 


1. Initializes memory. 

Loads the microcode update into the processor. 
Initializes the MTRRs. 

Enables the caches. 


Executes the CPUID instruction with a value of OH in the EAX register, then reads the EBX, ECX, and EDX 
registers to determine if the BSP is “GenuineIntel.” 


uo BR WN 


6. Executes the CPUID instruction with a value of 1H in the EAX register, then saves the values in the EAX, ECX, 
and EDX registers in a system configuration space in RAM for use later. 


Loads start-up code for the AP to execute into a 4-KByte page in the lower 1 MByte of memory. 


Switches to protected mode and ensures that the APIC address space is mapped to the strong uncacheable 
(UC) memory type. 
9. Determine the BSP’s APIC ID from the local APIC ID register (default is 0), the code snippet below is an 


example that applies to logical processors in a system whose local APIC units operate in xAPIC mode that APIC 
registers are accessed using memory mapped interface: 


MOV ESI, APIC_ID; Address of local APIC ID register 

MOV EAX, [ESI]; 

AND EAX, OFFOOOOOOH; Zero out all other bits except APIC ID 
MOV BOOT_ID, EAX; Save in memory 


Saves the APIC ID in the ACPI and/or MP tables and optionally in the system configuration space in RAM. 


10. Converts the base address of the 4-KByte page for the AP’s bootup code into 8-bit vector. The 8-bit vector 
defines the address of a 4-KByte page in the real-address mode address space (1-MByte space). For example, 
a vector of OBDH specifies a start-up memory address of OOOBDOOOH. 


11. Enables the local APIC by setting bit 8 of the APIC spurious vector register (SVR). 


MOV ESI, SVR; Address of SVR 

MOV EAX, [ESI]; 

OR EAX, APIC_ENABLED; Set bit 8 to enable (0 on reset) 
MOV [ESI], EAX; 


12. Sets up the LVT error handling entry by establishing an 8-bit vector for the APIC error handler. 


MOV ESI, LVT3; 

MOV EAX, [ESI]; 

AND EAX, FFFFFFOOH; Clear out previous vector. 

OR EAX, OOOO00xxH; xx is the 8-bit vector the APIC error handler. 
MOV [ESI], EAX; 


13. Initializes the Lock Semaphore variable VACANT to OOH. The APs use this semaphore to determine the order in 
which they execute BIOS AP initialization code. 


14. Performs the following operation to set up the BSP to detect the presence of APs in the system and the number 
of processors (within a finite duration, minimally 100 milliseconds): 


— Sets the value of the COUNT variable to 1. 


— Inthe AP BIOS initialization code, the AP will increment the COUNT variable to indicate its presence. The 
finite duration while waiting for the COUNT to be updated can be accomplished with a timer. When the timer 
expires, the BSP checks the value of the COUNT variable. If the timer expires and the COUNT variable has 
not been incremented, no APs are present or some error has occurred. 
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15. Broadcasts an INIT-SIPI-SIPI IPI sequence to the APs to wake them up and initialize them. If software knows 


how many logical processors it expects to wake up, it may choose to poll the COUNT variable. If the expected 
processors show up before the 100 millisecond timer expires, the timer can be canceled and skip to step 16. 

The left-hand-side of the procedure illustrated in Table 8-1 provides an algorithm when the expected processor 
count is unknown. The right-hand-side of Table 8-1 can be used when the expected processor count is known. 


Table 8-1. Broadcast INIT-SIPI-SIPI Sequence and Choice of Timeouts 


INIT-SIPI-SIPI when the expected processor count is unknown 


INIT-SIPI-SIPI when the expected processor count is known 


MOV ESI, ICR_LOW; Load address of ICR low dword into ESI. 
MOV EAX, 000C4500H; Load ICR encoding for broadcast INIT IPI 
; to all APs into EAX. 

MOV [ESI], EAX; Broadcast INIT IPI to all APs 

; 10-millisecond delay loop. 

MOV EAX, OOOC46XXH; Load ICR encoding for broadcast SIPI IP 
; to all APs into EAX, where xx is the vector computed in step 10. 
MOV [ESI], EAX; Broadcast SIPI IPI to all APs 

; 200-microsecond delay loop 

MOV [ESI], EAX; Broadcast second SIPI IPI to all APs 

;Waits for the timer interrupt until the timer expires 


MOV ESI, ICR_LOW; Load address of ICR low dword into ESI. 
MOV EAX, OO0C4500H; Load ICR encoding for broadcast INIT IPI 
; to all APs into EAX. 

MOV [ESI], EAX; Broadcast INIT IPI to all APs 

; 10-millisecond delay loop. 

MOV EAX, OOOC46XXH; Load ICR encoding for broadcast SIPI IP 
; to all APs into EAX, where xx is the vector computed in step 10. 
MOV [ESI], EAX; Broadcast SIPI IPI to all APs 

; 200 microsecond delay loop with check to see if COUNT has 

; reached the expected processor count. If COUNT reaches 

; expected processor count, cancel timer and go to step 16. 
MOV [ESI], EAX; Broadcast second SIPI IPI to all APs 

; Wait for the timer interrupt polling COUNT. If COUNT reaches 

; expected processor count, cancel timer and go to step 16. 

; If timer expires, go to step 16. 


16. Reads and evaluates the COUNT variable and establishes a processor count. 


17. If necessary, reconfigures the APIC and continues with the remaining system diagnostics as appropriate. 


8.4.4.2 Typical AP Initialization Sequence 


When an AP receives the SIPI, it begins executing BIOS AP initialization code at the vector encoded in the SIPI. The 
AP initialization code typically performs the following operations: 


1. Waits on the BIOS initialization Lock Semaphore. When control of the semaphore is attained, initialization 


continues. 


Loads the microcode update into the processor. 


Initializes the MTRRs (using the same mapping that was used for the BSP). 


2 
3 
4. Enables the cache. 
5 


Executes the CPUID instruction with a value of OH in the EAX register, then reads the EBX, ECX, and EDX 


registers to determine if the AP is “GenuineIntel.” 


6. Executes the CPUID instruction with a value of 1H in the EAX register, then saves the values in the EAX, ECX, 
and EDX registers in a system configuration space in RAM for use later. 


7. Switches to protected mode and ensures that the APIC address space is mapped to the strong uncacheable 


(UC) memory type. 


8. Determines the AP’s APIC ID from the local APIC ID register, and adds it to the MP and ACPI tables and 


optionally to the system configuration space in RAM. 


9. Initializes and configures the local APIC by setting bit 8 in the SVR register and setting up the LVT3 (error LVT) 
for error handling (as described in steps 9 and 10 in Section 8.4.4.1, “Typical BSP Initialization Sequence”). 
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10. Configures the APs SMI execution environment. (Each AP and the BSP must have a different SMBASE address. ) 
11. Increments the COUNT variable by 1. 
12. Releases the semaphore. 
13. Executes one of the following: 
— the CLI and HLT instructions (if MONITOR/MWAIT is not supported), or 
— the CLI, MONITOR and MWAIT sequence to enter a deep C-state. 
14. Waits for an INIT IPI. 


8.4.5 Identifying Logical Processors in an MP System 


After the BIOS has completed the MP initialization protocol, each logical processor can be uniquely identified by its 
local APIC ID. Software can access these APIC IDs in either of the following ways: 


¢ Read APIC ID for a local APIC — Code running on a logical processor can read APIC ID in one of two ways 
depending on the local APIC unit is operating in x2APIC mode (see Inte/® 64 Architecture x2APIC Specifi- 
cation)or in xAPIC mode: 


— Ifthe local APIC unit supports x2APIC and is operating in x2APIC mode, 32-bit APIC ID can be read by 
executing a RDMSR instruction to read the processor’s x2APIC ID register. This method is equivalent to 
executing CPUID leaf OBH described below. 


— Ifthe local APIC unit is operating in xAPIC mode, 8-bit APIC ID can be read by executing a MOV instruction 
to read the processor’s local APIC ID register (see Section 10.4.6, “Local APIC ID”). This is the ID to use for 
directing physical destination mode interrupts to the processor. 


° Read ACPI or MP table — As part of the MP initialization protocol, the BIOS creates an ACPI table and an MP 
table. These tables are defined in the Multiprocessor Specification Version 1.4 and provide software with a list 
of the processors in the system and their local APIC IDs. The format of the ACPI table is derived from the ACPI 
specification, which is an industry standard power management and platform configuration specification for MP 
systems. 


* Read Initial APIC ID (If the processor does not support CPUID leaf OBH) — An APIC ID is assigned to a logical 
processor during power up. This is the initial APIC ID reported by CPUID.1:EBX[31:24] and may be different 
from the current value read from the local APIC. The initial APIC ID can be used to determine the topological 
relationship between logical processors for multi-processor systems that do not support CPUID leaf OBH. 


Bits in the 8-bit initial APIC ID can be interpreted using several bit masks. Each bit mask can be used to extract 
an identifier to represent a hierarchical level of the multi-threading resource topology in an MP system (See 
Section 8.9.1, “Hierarchical Mapping of Shared Resources”). The initial APIC ID may consist of up to four bit- 
fields. In a non-clustered MP system, the field consists of up to three bit fields. 


¢ Read 32-bit APIC ID from CPUID leaf OBH (If the processor supports CPUID leaf OBH) — A unique APIC ID 
is assigned to a logical processor during power up. This APIC ID is reported by CPUID.OBH:EDX[31:0] as a 32- 
bit value. Use the 32-bit APIC ID and CPUID leaf OBH to determine the topological relationship between logical 
processors if the processor supports CPUID leaf OBH. 


Bits in the 32-bit x2APIC ID can be extracted into sub-fields using CPUID leaf OBH parameters. (See Section 
8.9.1, “Hierarchical Mapping of Shared Resources”). 


Figure 8-2 shows two examples of APIC ID bit fields in earlier single-core processors. In single-core Intel Xeon 
processors, the APIC ID assigned to a logical processor during power-up and initialization is 8 bits. Bits 2:1 forma 
2-bit physical package identifier (which can also be thought of as a socket identifier). In systems that configure 
physical processors in clusters, bits 4:3 form a 2-bit cluster ID. Bit 0 is used in the Intel Xeon processor MP to iden- 
tify the two logical processors within the package (see Section 8.9.3, “Hierarchical ID of Logical Processors in an MP 
System”). For Intel Xeon processors that do not support Intel Hyper-Threading Technology, bit 0 is always set to 0; 
for Intel Xeon processors supporting Intel Hyper-Threading Technology, bit 0 performs the same function as it does 
for Intel Xeon processor MP. 


For more recent multi-core processors, see Section 8.9.1, “Hierarchical Mapping of Shared Resources” for a 
complete description of the topological relationships between logical processors and bit field locations within an 
initial APIC ID across Intel 64 and IA-32 processor families. 


Vol. 3A 8-23 


MULTIPLE-PROCESSOR MANAGEMENT 


Note the number of bit fields and the width of bit-fields are dependent on processor and platform hardware capa- 
bilities. Software should determine these at runtime. When initial APIC IDs are assigned to logical processors, the 
value of APIC ID assigned to a logical processor will respect the bit-field boundaries corresponding core, physical 
package, etc. Additional examples of the bit fields in the initial APIC ID of multi-threading capable systems are 
shown in Section 8.9. 


APIC ID Format for Intel Xeon Processors that 
do not Support Intel Hyper-Threading Technology 


7 5 4 3 2 1 0 


Reserved | | 0 


Cluster _ 


Processor ID 


APIC ID Format for P6 Family Processors 
7 4 3 2 1 0 


Reserved | 


Cluster 
Processor ID 


Figure 8-2. Interpretation of APIC ID in Early MP Systems 


For P6 family processors, the APIC ID that is assigned to a processor during power-up and initialization is 4 bits 
(see Figure 8-2). Here, bits 0 and 1 form a 2-bit processor (or socket) identifier and bits 2 and 3 form a 2-bit cluster 
ID. 


8.5 INTEL® HYPER-THREADING TECHNOLOGY AND INTEL® MULTI-CORE 
TECHNOLOGY 


Intel Hyper-Threading Technology and Intel multi-core technology are extensions to Intel 64 and IA-32 architec- 
tures that enable a single physical processor to execute two or more separate code streams (called threads) 
concurrently. In Intel Hyper-Threading Technology, a single processor core provides two logical processors that 
share execution resources (see Section 8.7, “Intel® Hyper-Threading Technology Architecture”). In Intel multi-core 
technology, a physical processor package provides two or more processor cores. Both configurations require chip- 
sets and a BIOS that support the technologies. 


Software should not rely on processor names to determine whether a processor supports Intel Hyper-Threading 
Technology or Intel multi-core technology. Use the CPUID instruction to determine processor capability (see 
Section 8.6.2, “Initializing Multi-Core Processors”). 


8.6 DETECTING HARDWARE MULTI-THREADING SUPPORT AND TOPOLOGY 


Use the CPUID instruction to detect the presence of hardware multi-threading support in a physical processor. 
Hardware multi-threading can support several varieties of multigrade and/or Intel Hyper-Threading Technology. 
CPUID instruction provides several sets of parameter information to aid software enumerating topology informa- 
tion. The relevant topology enumeration parameters provided by CPUID include: 


¢ Hardware Multi-Threading feature flag (CPUID.1:EDX[28] = 1) — Indicates when set that the physical 
package is capable of supporting Intel Hyper-Threading Technology and/or multiple cores. 
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¢ Processor topology enumeration parameters for 8-bit APIC ID: 


— Addressable IDs for Logical processors in the same Package (CPUID.1:EBX[23:16]) — Indicates 
the maximum number of addressable ID for logical processors in a physical package. Within a physical 
package, there may be addressable IDs that are not occupied by any logical processors. This parameter 
does not represents the hardware capability of the physical processor.® 


* Addressable IDs for processor cores in the same Package’ (CPUID.(EAX=4, ECX=0°):EAX[31:26] + 
1 = Y) — Indicates the maximum number of addressable IDs attributable to processor cores (Y) in the physical 
package. 


¢ Extended Processor Topology Enumeration parameters for 32-bit APIC ID: Intel 64 processors 
supporting CPUID leaf OBH will assign unique APIC IDs to each logical processor in the system. CPUID leaf OBH 
reports the 32-bit APIC ID and provide topology enumeration parameters. See CPUID instruction reference 
pages in Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A. 


The CPUID feature flag may indicate support for hardware multi-threading when only one logical processor avail- 
able in the package. In this case, the decimal value represented by bits 16 through 23 in the EBX register will have 
a value of 1. 


Software should note that the number of logical processors enabled by system software may be less than the value 
of “Addressable IDs for Logical processors”. Similarly, the number of cores enabled by system software may be less 
than the value of “Addressable IDs for processor cores”. 


Software can detect the availability of the CPUID extended topology enumeration leaf (OBH) by performing two 
steps: 


¢ Check maximum input value for basic CPUID information by executing CPUID with EAX= 0. If CPUID.OH:EAX is 
greater than or equal or 11 (OBH), then proceed to next step, 


° Check CPUID.EAX=0BH, ECX=0H:EBxX is non-zero. 


If both of the above conditions are true, extended topology enumeration leaf is available. Note the presence of 
CPUID leaf OBH in a processor does not guarantee support that the local APIC supports x2APIC. If 
CPUID.(EAX=0BH, ECX=0H):EBX returns zero and maximum input value for basic CPUID information is greater 
than OBH, then CPUID.OBH leaf is not supported on that processor. 


8.6.1 Initializing Processors Supporting Hyper-Threading Technology 


The initialization process for an MP system that contains processors supporting Intel Hyper-Threading Technology 
is the same as for conventional MP systems (see Section 8.4, “Multiple-Processor (MP) Initialization”). One logical 
processor in the system is selected as the BSP and other processors (or logical processors) are designated as APs. 
The initialization process is identical to that described in Section 8.4.3, “MP Initialization Protocol Algorithm for MP 
Systems,” and Section 8.4.4, “MP Initialization Example.” 


During initialization, each logical processor is assigned an APIC ID that is stored in the local APIC ID register for 
each logical processor. If two or more processors supporting Intel Hyper-Threading Technology are present, each 
logical processor on the system bus is assigned a unique ID (see Section 8.9.3, “Hierarchical ID of Logical Proces- 
sors in an MP System”). Once logical processors have APIC IDs, software communicates with them by sending APIC 
IPI messages. 


6. Operating system and BIOS may implement features that reduce the number of logical processors available in a platform to applica- 
tions at runtime to less than the number of physical packages times the number of hardware-capable logical processors per pack- 
age. 

7. Software must check CPUID for its support of leaf 4 when implementing support for multi-core. If CPUID leaf 4 is not available at 
runtime, software should handle the situation as if there is only one core per package. 

8. Maximum number of cores in the physical package must be queried by executing CPUID with EAX=4 and a valid ECX input value. 
Valid ECX input values start from O. 
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8.6.2 Initializing Multi-Core Processors 


The initialization process for an MP system that contains multi-core Intel 64 or IA-32 processors is the same as for 
conventional MP systems (see Section 8.4, “Multiple-Processor (MP) Initialization”). A logical processor in one core 
is selected as the BSP; other logical processors are designated as APs. 


During initialization, each logical processor is assigned an APIC ID. Once logical processors have APIC IDs, software 
may communicate with them by sending APIC IPI messages. 


8.6.3 Executing Multiple Threads on an Intel® 64 or IA-32 Processor Supporting Hardware 
Multi-Threading 


Upon completing the operating system boot-up procedure, the bootstrap processor (BSP) executes operating 
system code. Other logical processors are placed in the halt state. To execute a code stream (thread) on a halted 
logical processor, the operating system issues an interprocessor interrupt (IPI) addressed to the halted logical 
processor. In response to the IPI, the processor wakes up and begins executing the code identified by the vector 
received as part of the IPI. 


To manage execution of multiple threads on logical processors, an operating system can use conventional 
symmetric multiprocessing (SMP) techniques. For example, the operating-system can use a time-slice or load 
balancing mechanism to periodically interrupt each of the active logical processors. Upon interrupting a logical 
processor, the operating system checks its run queue for a thread waiting to be executed and dispatches the thread 
to the interrupted logical processor. 


8.6.4 Handling Interrupts on an IA-32 Processor Supporting Hardware Multi-Threading 


Interrupts are handled on processors supporting Intel Hyper-Threading Technology as they are on conventional MP 
systems. External interrupts are received by the I/O APIC, which distributes them as interrupt messages to specific 
logical processors (see Figure 8-3). 


Logical processors can also send IPIs to other logical processors by writing to the ICR register of its local APIC (see 
Section 10.6, “Issuing Interprocessor Interrupts”). This also applies to dual-core processors. 


8-26 Vol. 3A 


MULTIPLE-PROCESSOR MANAGEMENT 


Intel Processor with Intel 
Hyper-Threading Technology Hyper-Threading Technology 


Intel Processor with Intel 


Logical 
Processor 0 


Logical Logical 
Processor 1 Processor 0 


Logical 
Processor 1 


Processor Core 


Processor Core 


Local APIC 


Local APIC Local APIC 


Local APIC 


Bus Interface 


Bus Interface 


IPIs 


A Interrupt 


Messages Ps 


System Chip Set 


Interrupt 
Messages 


A Interrupt Messages 


~<— External 
ciate ~<—_ Interrupts 


Figure 8-3. Local APICs and I/O APIC in MP System Supporting Intel HT Technology 


8.7 INTEL® HYPER-THREADING TECHNOLOGY ARCHITECTURE 


Figure 8-4 shows a generalized view of an Intel processor supporting Intel Hyper-Threading Technology, using the 
original Intel Xeon processor MP as an example. This implementation of the Intel Hyper-Threading Technology 

consists of two logical processors (each represented by a separate architectural state) which share the processor's 
execution engine and the bus interface. Each logical processor also has its own advanced programmable interrupt 


controller (APIC). 
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Figure 8-4. IA-32 Processor with Two Logical Processors Supporting Intel HT Technology 


8.7.1 State of the Logical Processors 


The following features are part of the architectural state of logical processors within Intel 64 or IA-32 processors 
supporting Intel Hyper-Threading Technology. The features can be subdivided into three groups: 


* Duplicated for each logical processor 

¢ Shared by logical processors in a physical processor 

¢ Shared or duplicated, depending on the implementation 

The following features are duplicated for each logical processor: 

® General purpose registers (EAX, EBX, ECX, EDX, ESI, EDI, ESP, and EBP) 
* Segment registers (CS, DS, SS, ES, FS, and GS) 


¢ EFLAGS and EIP registers. Note that the CS and EIP/RIP registers for each logical processor point to the 
instruction stream for the thread being executed by the logical processor. 


° x87 FPU registers (STO through ST7, status word, control word, tag word, data operand pointer, and instruction 
pointer) 


¢ MMX registers (MMO through MM7) 

¢ XMM registers (XMMO through XMM7) and the MXCSR register 

* Control registers and system table pointer registers (GDTR, LDTR, IDTR, task register) 

* Debug registers (DRO, DR1, DR2, DR3, DR6, DR7) and the debug control MSRs 

¢ Machine check global status (IA32_MCG_STATUS) and machine check capability (IA32_MCG_CAP) MSRs 
¢ Thermal clock modulation and ACPI Power management control MSRs 

° Time stamp counter MSRs 

° Most of the other MSR registers, including the page attribute table (PAT). See the exceptions below. 

° Local APIC registers. 


e Additional general purpose registers (R8-R15), XMM registers (XMM8-XMM15), control register, IA32_EFER on 
Intel 64 processors. 


The following features are shared by logical processors: 

¢ Memory type range registers (MTRRs) 

Whether the following features are shared or duplicated is implementation-specific: 
¢ TA32_MISC_ENABLE MSR (MSR address 1A0H) 
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¢ Machine check architecture (MCA) MSRs (except for the IA32_MCG_STATUS and IA32_MCG_CAP MSRs) 
¢ Performance monitoring control and counter MSRs 


8.7.2 APIC Functionality 


When a processor supporting Intel Hyper-Threading Technology support is initialized, each logical processor is 
assigned a local APIC ID (see Table 10-1). The local APIC ID serves as an ID for the logical processor and is stored 
in the logical processor’s APIC ID register. If two or more processors supporting Intel Hyper-Threading Technology 
are present in a dual processor (DP) or MP system, each logical processor on the system bus is assigned a unique 
local APIC ID (see Section 8.9.3, “Hierarchical ID of Logical Processors in an MP System”). 


Software communicates with local processors using the APIC’s interprocessor interrupt (IPI) messaging facility. 
Setup and programming for APICs is identical in processors that support and do not support Intel Hyper-Threading 
Technology. See Chapter 10, “Advanced Programmable Interrupt Controller (APIC),” for a detailed discussion. 


8.7.3 Memory Type Range Registers (MTRR) 


MTRRs in a processor supporting Intel Hyper-Threading Technology are shared by logical processors. When one 
logical processor updates the setting of the MTRRs, settings are automatically shared with the other logical proces- 
sors in the same physical package. 


The architectures require that all MP systems based on Intel 64 and IA-32 processors (this includes logical proces- 
sors) must use an identical MTRR memory map. This gives software a consistent view of memory, independent of 
the processor on which it is running. See Section 11.11, “Memory Type Range Registers (MTRRs),” for information 
on setting up MTRRs. 


8.7.4 Page Attribute Table (PAT) 


Each logical processor has its own PAT MSR (IA32_PAT). However, as described in Section 11.12, “Page Attribute 
Table (PAT),” the PAT MSR settings must be the same for all processors in a system, including the logical proces- 
sors. 


8.7.5 Machine Check Architecture 


In the Intel HT Technology context as implemented by processors based on Intel NetBurst® microarchitecture, all 
of the machine check architecture (MCA) MSRs (except for the IA32_MCG_STATUS and IA32_MCG_CAP MSRs) are 
duplicated for each logical processor. This permits logical processors to initialize, configure, query, and handle 
machine-check exceptions simultaneously within the same physical processor. The design is compatible with 
machine check exception handlers that follow the guidelines given in Chapter 15, “Machine-Check Architecture.” 


The IA32_MCG_STATUS MSR is duplicated for each logical processor so that its machine check in progress bit field 
(MCIP) can be used to detect recursion on the part of MCA handlers. In addition, the MSR allows each logical 
processor to determine that a machine-check exception is in progress independent of the actions of another logical 
processor in the same physical package. 


Because the logical processors within a physical package are tightly coupled with respect to shared hardware 
resources, both logical processors are notified of machine check errors that occur within a given physical processor. 
If machine-check exceptions are enabled when a fatal error is reported, all the logical processors within a physical 
package are dispatched to the machine-check exception handler. If machine-check exceptions are disabled, the 
logical processors enter the shutdown state and assert the IERR# signal. 


When enabling machine-check exceptions, the MCE flag in control register CR4 should be set for each logical 
processor. 


On Intel Atom family processors that support Intel Hyper-Threading Technology, the MCA facilities are shared 
between all logical processors on the same processor core. 
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8.7.6 Debug Registers and Extensions 


Each logical processor has its own set of debug registers (DRO, DR1, DR2, DR3, DR6, DR7) and its own debug 
control MSR. These can be set to control and record debug information for each logical processor independently. 
Each logical processor also has its own last branch records (LBR) stack. 


8.7.7 Performance Monitoring Counters 


Performance counters and their companion control MSRs are shared between the logical processors within a 
processor core for processors based on Intel NetBurst microarchitecture. As a result, software must manage the 
use of these resources. The performance counter interrupts, events, and precise event monitoring support can be 
set up and allocated on a per thread (per logical processor) basis. 


See Section 18.6.4, “Performance Monitoring and Intel Hyper-Threading Technology in Processors Based on Intel 
NetBurst® Microarchitecture,” for a discussion of performance monitoring in the Intel Xeon processor MP. 


In Intel Atom processor family that support Intel Hyper-Threading Technology, the performance counters (general- 
purpose and fixed-function counters) and their companion control MSRs are duplicated for each logical processor. 


8.7.8 IA32_MISC_ENABLE MSR 


The IA32_MISC_ENABLE MSR (MSR address 1A0H) is generally shared between the logical processors in a 
processor core supporting Intel Hyper-Threading Technology. However, some bit fields within IA32_MISC_ENABLE 
MSR may be duplicated per logical processor. The partition of shared or duplicated bit fields within 
IA32_MISC_ENABLE is implementation dependent. Software should program duplicated fields carefully on all 
logical processors in the system to ensure consistent behavior. 


8.7.9 Memory Ordering 


The logical processors in an Intel 64 or IA-32 processor supporting Intel Hyper-Threading Technology obey the 
same rules for memory ordering as Intel 64 or IA-32 processors without Intel HT Technology (see Section 8.2, 
“Memory Ordering”). Each logical processor uses a processor-ordered memory model that can be further defined 
as “write-ordered with store buffer forwarding.” All mechanisms for strengthening or weakening the memory- 
ordering model to handle special programming situations apply to each logical processor. 


8.7.10 Serializing Instructions 


As a general rule, when a logical processor in a processor supporting Intel Hyper-Threading Technology executes a 
serializing instruction, only that logical processor is affected by the operation. An exception to this rule is the execu- 
tion of the WBINVD, INVD, and WRMSR instructions; and the MOV CR instruction when the state of the CD flag in 
control register CRO is modified. Here, both logical processors are serialized. 


8.7.11. Microcode Update Resources 


In an Intel processor supporting Intel Hyper-Threading Technology, the microcode update facilities are shared 
between the logical processors; either logical processor can initiate an update. Each logical processor has its own 
BIOS signature MSR (IA32_BIOS_SIGN_ID at MSR address 8BH). When a logical processor performs an update for 
the physical processor, the IA32_BIOS_SIGN_ID MSRs for resident logical processors are updated with identical 
information. If logical processors initiate an update simultaneously, the processor core provides the necessary 
synchronization needed to ensure that only one update is performed at a time. 
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NOTE 


Some processors (prior to the introduction of Intel 64 Architecture and based on Intel NetBurst 
microarchitecture) do not support simultaneous loading of microcode update to the sibling logical 
processors in the same core. All other processors support logical processors initiating an update 
simultaneously. Intel recommends a common approach that the microcode loader use the 
sequential technique described in Section 9.11.6.3. 


8.7.12 Self Modifying Code 


Intel processors supporting Intel Hyper-Threading Technology support self-modifying code, where data writes 
modify instructions cached or currently in flight. They also support cross-modifying code, where on an MP system 
writes generated by one processor modify instructions cached or currently in flight on another. See Section 8.1.3, 
“Handling Self- and Cross-Modifying Code,” for a description of the requirements for self- and cross-modifying code 
in an IA-32 processor. 


8.7.13. Implementation-Specific Intel HT Technology Facilities 


The following non-architectural facilities are implementation-specific in IA-32 processors supporting Intel Hyper- 
Threading Technology: 


° Caches 
¢ Translation lookaside buffers (TLBs) 
¢ Thermal monitoring facilities 


The Intel Xeon processor MP implementation is described in the following sections. 


8.7.13.1 Processor Caches 


For processors supporting Intel Hyper-Threading Technology, the caches are shared. Any cache manipulation 
instruction that is executed on one logical processor has a global effect on the cache hierarchy of the physical 
processor. Note the following: 


¢ WBINVD instruction — The entire cache hierarchy is invalidated after modified data is written back to 
memory. All logical processors are stopped from executing until after the write-back and invalidate operation is 
completed. A special bus cycle is sent to all caching agents. The amount of time or cycles for WBINVD to 
complete will vary due to the size of different cache hierarchies and other factors. As a consequence, the use of 
the WBINVD instruction can have an impact on interrupt/event response time. 


¢ INVD instruction — The entire cache hierarchy is invalidated without writing back modified data to memory. 
All logical processors are stopped from executing until after the invalidate operation is completed. A special bus 
cycle is sent to all caching agents. 


¢ CLFLUSH and CLFLUSHOPT instructions — The specified cache line is invalidated from the cache hierarchy 
after any modified data is written back to memory and a bus cycle is sent to all caching agents, regardless of 
which logical processor caused the cache line to be filled. 


¢ CD flag in control register CRO — Each logical processor has its own CRO control register, and thus its own 
CD flag in CRO. The CD flags for the two logical processors are ORed together, such that when any logical 
processor sets its CD flag, the entire cache is nominally disabled. 


8.7.13.2 Processor Translation Lookaside Buffers (TLBs) 


In processors supporting Intel Hyper-Threading Technology, data cache TLBs are shared. The instruction cache TLB 
may be duplicated or shared in each logical processor, depending on implementation specifics of different 
processor families. 


Entries in the TLBs are tagged with an ID that indicates the logical processor that initiated the translation. This tag 
applies even for translations that are marked global using the page-global feature for memory paging. See Section 
4.10, “Caching Translation Information,” for information about global translations. 
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When a logical processor performs a TLB invalidation operation, only the TLB entries that are tagged for that logical 
processor are guaranteed to be flushed. This protocol applies to all TLB invalidation operations, including writes to 
control registers CR3 and CR4 and uses of the INVLPG instruction. 


8.7.13.3. Thermal Monitor 


In a processor that supports Intel Hyper-Threading Technology, logical processors share the catastrophic shutdown 
detector and the automatic thermal monitoring mechanism (see Section 14.8, “Thermal Monitoring and Protec- 
tion”). Sharing results in the following behavior: 


e If the processor’s core temperature rises above the preset catastrophic shutdown temperature, the processor 
core halts execution, which causes both logical processors to stop execution. 


¢ When the processor's core temperature rises above the preset automatic thermal monitor trip temperature, the 
frequency of the processor core is automatically modulated, which effects the execution speed of both logical 
processors. 


For software controlled clock modulation, each logical processor has its own IA32_CLOCK_MODULATION MSR, 
allowing clock modulation to be enabled or disabled on a logical processor basis. Typically, if software controlled 
clock modulation is going to be used, the feature must be enabled for all the logical processors within a physical 
processor and the modulation duty cycle must be set to the same value for each logical processor. If the duty cycle 
values differ between the logical processors, the processor clock will be modulated at the highest duty cycle 
selected. 


8.7.13.4 External Signal Compatibility 


This section describes the constraints on external signals received through the pins of a processor supporting Intel 
Hyper-Threading Technology and how these signals are shared between its logical processors. 


° STPCLK# — A single STPCLK# pin is provided on the physical package of the Intel Xeon processor MP. External 
control logic uses this pin for power management within the system. When the STPCLK# signal is asserted, the 
processor core transitions to the stop-grant state, where instruction execution is halted but the processor core 
continues to respond to snoop transactions. Regardless of whether the logical processors are active or halted 
when the STPCLK# signal is asserted, execution is stopped on both logical processors and neither will respond 
to interrupts. 


In MP systems, the STPCLK# pins on all physical processors are generally tied together. As a result this signal 
affects all the logical processors within the system simultaneously. 


¢ LINTO and LINT1 pins — A processor supporting Intel Hyper-Threading Technology has only one set of LINTO 
and LINT1 pins, which are shared between the logical processors. When one of these pins is asserted, both 
logical processors respond unless the pin has been masked in the APIC local vector tables for one or both of the 
logical processors. 


Typically in MP systems, the LINTO and LINT1 pins are not used to deliver interrupts to the logical processors. 
Instead all interrupts are delivered to the local processors through the I/O APIC. 


* A20M# pin — On an IA-32 processor, the A20M# pin is typically provided for compatibility with the Intel 286 
processor. Asserting this pin causes bit 20 of the physical address to be masked (forced to zero) for all external 
bus memory accesses. Processors supporting Intel Hyper-Threading Technology provide one A20M# pin, which 
affects the operation of both logical processors within the physical processor. 


The functionality of AZ0M# is used primarily by older operating systems and not used by modern operating 
systems. On newer Intel 64 processors, A20M# may be absent. 


8.8 MULTI-CORE ARCHITECTURE 


This section describes the architecture of Intel 64 and IA-32 processors supporting dual-core and quad-core tech- 
nology. The discussion is applicable to the Intel Pentium processor Extreme Edition, Pentium D, Intel Core Duo, 
Intel Core 2 Duo, Dual-core Intel Xeon processor, Intel Core 2 Quad processors, and quad-core Intel Xeon proces- 
sors. Features vary across different microarchitectures and are detectable using CPUID. 
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In general, each processor core has dedicated microarchitectural resources identical to a single-processor imple- 
mentation of the underlying microarchitecture without hardware multi-threading capability. Each logical processor 
in a dual-core processor (whether supporting Intel Hyper-Threading Technology or not) has its own APIC function- 
ality, PAT, machine check architecture, debug registers and extensions. Each logical processor handles serialization 
instructions or self-modifying code on its own. Memory order is handled the same way as in Intel Hyper-Threading 
Technology. 


The topology of the cache hierarchy (with respect to whether a given cache level is shared by one or more 
processor cores or by all logical processors in the physical package) depends on the processor implementation. 
Software must use the deterministic cache parameter leaf of CPUID instruction to discover the cache-sharing 
topology between the logical processors in a multi-threading environment. 


8.8.1 Logical Processor Support 


The topological composition of processor cores and logical processors in a multi-core processor can be discovered 
using CPUID. Within each processor core, one or more logical processors may be available. 


System software must follow the requirement MP initialization sequences (see Section 8.4, “Multiple-Processor 
(MP) Initialization”) to recognize and enable logical processors. At runtime, software can enumerate those logical 
processors enabled by system software to identify the topological relationships between these logical processors. 
(See Section 8.9.5, “Identifying Topological Relationships in a MP System”). 


8.8.2 Memory Type Range Registers (MTRR) 


MTRR is shared between two logical processors sharing a processor core if the physical processor supports Intel 
Hyper-Threading Technology. MTRR is not shared between logical processors located in different cores or different 
physical packages. 


The Intel 64 and IA-32 architectures require that all logical processors in an MP system use an identical MTRR 
memory map. This gives software a consistent view of memory, independent of the processor on which it is 
running. 


See Section 11.11, “Memory Type Range Registers (MTRRs).” 


8.8.3 Performance Monitoring Counters 


Performance counters and their companion control MSRs are shared between two logical processors sharing a 
processor core if the processor core supports Intel Hyper-Threading Technology and is based on Intel NetBurst 
microarchitecture. They are not shared between logical processors in different cores or different physical packages. 
As a result, software must manage the use of these resources, based on the topology of performance monitoring 
resources. Performance counter interrupts, events, and precise event monitoring support can be set up and allo- 
cated on a per thread (per logical processor) basis. 


See Section 18.6.4, “Performance Monitoring and Intel Hyper-Threading Technology in Processors Based on Intel 
NetBurst® Microarchitecture.” 


8.8.4 IA32_MISC_ENABLE MSR 


Some bit fields in IA32_MISC_ENABLE MSR (MSR address 1A0H) may be shared between two logical processors 
sharing a processor core, or may be shared between different cores in a physical processor. See Chapter 2, “Model- 
Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 


8.8.5 Microcode Update Resources 


Microcode update facilities are shared between two logical processors sharing a processor core if the physical 
package supports Intel Hyper-Threading Technology. They are not shared between logical processors in different 
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cores or different physical packages. Either logical processor that has access to the microcode update facility can 
initiate an update. 


Each logical processor has its own BIOS signature MSR (IA32_BIOS_SIGN_ID at MSR address 8BH). When a logical 
processor performs an update for the physical processor, the IA32_BIOS_SIGN_ID MSRs for resident logical 
processors are updated with identical information. 


All microcode update steps during processor initialization should use the same update data on all cores in all phys- 
ical packages of the same stepping. Any subsequent microcode update must apply consistent update data to all 
cores in all physical packages of the same stepping. If the processor detects an attempt to load an older microcode 
update when a newer microcode update had previously been loaded, it may reject the older update to stay with the 
newer update. 


NOTE 


Some processors (prior to the introduction of Intel 64 Architecture and based on Intel NetBurst 
microarchitecture) do not support simultaneous loading of microcode update to the sibling logical 
processors in the same core. All other processors support logical processors initiating an update 
simultaneously. Intel recommends a common approach that the microcode loader use the 
sequential technique described in Section 9.11.6.3. 


8.9 PROGRAMMING CONSIDERATIONS FOR HARDWARE MULTI-THREADING 
CAPABLE PROCESSORS 


In a multi-threading environment, there may be certain hardware resources that are physically shared at some 
level of the hardware topology. In the multi-processor systems, typically bus and memory sub-systems are physi- 
cally shared between multiple sockets. Within a hardware multi-threading capable processors, certain resources 
are provided for each processor core, while other resources may be provided for each logical processors (see 
Section 8.7, “Intel® Hyper-Threading Technology Architecture,” and Section 8.8, “Multi-Core Architecture”). 


From a software programming perspective, control transfer of processor operation is managed at the granularity of 
logical processor (operating systems dispatch a runnable task by allocating an available logical processor on the 
platform). To manage the topology of shared resources in a multi-threading environment, it may be useful for soft- 
ware to understand and manage resources that are shared by more than one logical processors. 


8.9.1 Hierarchical Mapping of Shared Resources 


The APIC_ID value associated with each logical processor in a multi-processor system is unique (see Section 8.6, 
“Detecting Hardware Multi-Threading Support and Topology”). This 8-bit or 32-bit value can be decomposed into 
sub-fields, where each sub-field corresponds a hierarchical level of the topological mapping of hardware resources. 


The decomposition of an APIC_ID may consist of several sub fields representing the topology within a physical 
processor package, the higher-order bits of an APIC ID may also be used by cluster vendors to represent the 
topology of cluster nodes of each coherent multiprocessor systems: 


* Cluster — Some multi-threading environments consists of multiple clusters of multi-processor systems. The 
CLUSTER_ID sub-field is usually supported by vendor firmware to distinguish different clusters. For non- 
clustered systems, CLUSTER_ID is usually 0 and system topology is reduced. 


¢ Package — A physical processor package mates with a socket. A package may contain one or more software 
visible die. The PACKAGE_ID sub-field distinguishes different physical packages within a cluster. 


* Die — A software-visible chip inside a package. The DIE_ID sub-field distinguishes different die within a 
package. If there are no software visible die, the width of this bit field is 0. 


° Tile — A set of cores, possibly within modules, that share certain resources. The TILE_ID sub-field distin- 
guishes different tiles. If there are no software visible tiles, the width of this bit field is 0. 


° Module — A set of cores that share certain resources. The MODULE_ID sub-field distinguishes different 
modules. If there are no software visible modules, the width of this bit field is 0. 
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¢ Core — Processor cores may be contained within modules, within tiles, on software-visible die, or appear 
directly at the package level. The CORE_ID sub-field distinguishes processor cores. For a single-core processor, 
the width of this bit field is 0. 


¢ SMT — A processor core provides one or more logical processors sharing execution resources. The SMT_ID 
sub-field distinguishes logical processors in a core. The width of this bit field is non-zero if a processor core 
provides more than one logical processors. 


SMT and CORE sub-fields are bit-wise contiguous in the APIC_ID field (see Figure 8-5). 


X=31 if x2APIC supported, else X=7 
X 0 


Reserved 


Cluster ID —— 
Package ID ——————_— 
Die ID 
Tile ID 
Module ID 
Core ID 
SMT ID 


Figure 8-5. Generalized Seven Level Interpretation of the APIC ID 


If the processor supports CPUID leaf OBH and leaf 1FH, the 32-bit APIC ID can represent cluster plus several levels 
of topology within the physical processor package. The exact number of hierarchical levels within a physical 
processor package must be enumerated through CPUID leaf OBH and leaf 1FH. Common processor families may 
employ topology similar to that represented by 8-bit Initial APIC ID. In general, CPUID leaf OBH and leaf 1FH can 
support a topology enumeration algorithm that decompose a 32-bit APIC ID into more than four sub-fields (see 
Figure 8-6). 


NOTE 


CPUID leaf OBH and leaf 1FH can have differences in the number of level types reported (CPUID leaf 
1FH defines additional level types). If the processor supports CPUID leaf 1FH, usage of this leaf is 
preferred over leaf OBH. CPUID leaf OBH is available for legacy compatibility going forward. 


The width of each sub-field depends on hardware and software configurations. Field widths can be determined at 
runtime using the algorithm discussed below (Example 8-16 through Example 8-21). 


Figure 7-6 depicts the relationships of three of the hierarchical sub-fields in a hypothetical MP system. The value of 
valid APIC_IDs need not be contiguous across package boundary or core boundaries. 
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PACKAGE 31 0 


Q Reserved | | 


CORE Cluster ID 
SMT Package ID 
QID 
RID 


CORE ID 
SMT ID 


Physical Processor Topology 32-bit APIC ID Composition 


Figure 8-6. Conceptual Six-Level Topology and 32-bit APIC ID Composition 


8.9.2 Hierarchical Mapping of CPUID Extended Topology Leaf 


CPUID leaf OBH and leaf 1FH provide enumeration parameters for software to identify each hierarchy of the 
processor topology in a deterministic manner. Each hierarchical level of the topology starting from the SMT level is 
represented numerically by a sub-leaf index within the CPUID OBH leaf and 1FH leaf. Each level of the topology is 
mapped to a sub-field in the APIC ID, following the general relationship depicted in Figure 8-6. This mechanism 
allows software to query the exact number of levels within a physical processor package and the bit-width of each 
sub-field of x2APIC ID directly. For example, 


° Starting from sub-leaf index 0 and incrementing ECX until CPUID.(EAX=OBH or 1FH, ECX=N):ECX[15:8] 
returns an invalid “level type” encoding. The number of levels within the physical processor package is “N” 


(excluding PACKAGE). Using Figure 8-6 as an example, CPUID.(EAX=OBH or 1FH, ECX=4):ECX[15:8] will 
report OOH, indicating sub leaf 04H is invalid. This is also depicted by a pseudo code example: 


Example 8-16. Number of Levels Below the Physical Processor Package 


Byte type = 1; 
s=0; 
While ( type ) { 
EAX = OBH or 1FH; // query each sub leaf of CPUID leaf OBH or 1FH; CPUID leaf 1FH is preferred over leaf OBH if available 


ECX = s; 
CPUID; 
type = ECX[15:8]; // examine level type encoding 
s t+; 
} 
N = ECX[7:0]; 


Sub-leaf index 0 (ECX= 0 as input) provides enumeration parameters to extract the SMT sub-field of x2APIC 
ID. If EAX = OBH or 1FH, and ECX =0 is specified as input when executing CPUID, CPUID.(EAX=O0BH or 1FH, 
ECX=0):EAX[4:0] reports a value (a right-shift count) that allow software to extract part of x2APIC ID to 
distinguish the next higher topological entities above the SMT level. This value also corresponds to the bit-width 
of the sub-field of x2APIC ID corresponding the hierarchical level with sub-leaf index 0. 


¢ For each subsequent higher sub-leaf index m, CPUID.(EAX=0BH or 1FH, ECX=m):EAX[4:0] reports the right- 
shift count that will allow software to extract part of x2APIC ID to distinguish higher-level topological entities. 


This means the right-shift value at of sub-leaf m, corresponds to the least significant (m+1) subfields of the 32- 
bit x2APIC ID. 
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Example 8-17. BitwWidth Determination of x2APIC ID Subfields 


For m=0,m<N,m ++; 
{ cumulative_width[m] = CPUID.(EAX=OBH or 1FH, ECX= m): EAX[4:0]; } 
BitWidth[O] = cumulative_width[O]; 
Form=1,m<N,m++; 
BitWidth[m] = cumulative_width[m] - cumulative_width[m-1]; 


NOTE 


CPUID leaf 1FH is a preferred superset to leaf OBH. Leaf 1FH defines additional level types, and it 
must be parsed by an algorithm that can handle the addition of future level types. 


Previously, only the following encoding of hierarchical level types were defined: 0 (invalid), 1 (SMT), and 2 (core). 
With the additional hierarchical level types available (see Section 8.9.1, “Hierarchical Mapping of Shared 
Resources” and Figure 8-5, “Generalized Seven Level Interpretation of the APIC ID” ) software must not assume 
any “level type” encoding value to be related to any sub-leaf index, except sub-leaf 0. 


Example 8-18. Support Routines for Identifying Package, Die, Core and Logical Processors from 32-bit x2APIC ID 


a. Derive the extraction bitmask for logical processors in a processor core and associated mask offset for different 
cores. 
// 


// This example shows how to enumerate CPU topology level types (level types may or may not be known/supported by the software) 
1 

// Below is the list of sample level types used in the example. 

// Refer to the CPUID Leaf 1FH definition for the actual level type numbers: “V2 Extended Topology Enumeration Leaf” . 

// 

11 SMT 

// CORE 

// MODULE 

// TILE 

// DIE 

// PACKAGE 

// 

// The example shows how to identify and derive the extraction bitmask for the levels with identify type SMT/CORE/DIE/PACKAGE 
// 


int DeriveSMT_Mask_Offsets (void) 


i 
IF (\HWMTSupported()) return -1; 
execute cpuid with EAX = OBH or 1FH, ECX = 0; 
IF (returned level type encoding in EXC[15:8] does not match SMT) return -1; 
Mask_SMT_shift = EAX[4:0]; //# bits shift right of APIC ID to distinguish different cores 
SMT_MASK =~( (-1) << Mask_SMT_shift); //shift left to derive extraction bitmask for SMT_ID 
return 0; 
} 
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b. Derive the extraction bitmask for processor cores in a physical processor package and associated mask offset for 
different packages. 


int DeriveCore_Mask_Offsets (void) 
{ 
IF (\HWMTSupported()) return -1; 
execute cpuid with EAX = OBH or 1FH, ECX = 0; 
WHILE( ECX[15:8] ) { //level type encoding is valid 
Mask_last_known_shift = EAX[4:0] 
IF (returned level type encoding in ECX[15:8] matches CORE) { 
Mask_Core_shift = EAX[4:0]; 
} 
ELSE IF (returned level type encoding in ECX[15:8] matches DIE { 
Mask_Die_shift = EAX[4:0]; 
} 
// 
// Keep enumerating. Check if the next level is the desired level and if not, keep enumerating until you reach a known level 
// or the invalid level (“O" level type). If there are more levels between DIE and PACKAGE, the unknown levels will be ignored 
// and treated as an extension of the last known level (i.e., DIE in this case). 
// 


ECX++; 
execute cpuid with EAX = OBH or 1FH; 
} 


COREPlusSMT_MASK = ~( (-1) << Mask_Core_shift); 
DIEPlusCORE_MASK = ~( (-1) << Mask_Die_shift); 


// 

// Treat levels between DIE and physical package as an extension of DIE for software choosing not to implement or recognize 
// these unknown levels. 

// 


CORE_MASK = COREPIusSMT_MASK * SMT_MASK; 
DIE_MASK = DIEPlusCORE_MASK * COREPlusSMT_MASK; 
PACKAGE_MASK = (-1) << Mask_last_known_shift; 


return -1; 


8.9.3 Hierarchical ID of Logical Processors in an MP System 


For Intel 64 and IA-32 processors, system hardware establishes an 8-bit initial APIC ID (or 32-bit APIC ID if the 
processor supports CPUID leaf OBH) that is unique for each logical processor following power-up or RESET (see 
Section 8.6.1). Each logical processor on the system is allocated an initial APIC ID. BIOS may implement features 
that tell the OS to support less than the total number of logical processors on the system bus. Those logical proces- 
sors that are not available to applications at runtime are halted during the OS boot process. As a result, the number 
valid local APIC_IDs that can be queried by affinitizing-current-thread-context (See Example 8-23) is limited to the 
number of logical processors enabled at runtime by the OS boot process. 


Table 8-2 shows an example of the 8-bit APIC IDs that are initially reported for logical processors in a system with 
four Intel Xeon MP processors that support Intel Hyper-Threading Technology (a total of 8 logical processors, each 
physical package has two processor cores and supports Intel Hyper-Threading Technology). Of the two logical 
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processors within a Intel Xeon processor MP, logical processor 0 is designated the primary logical processor and 
logical processor 1 as the secondary logical processor. 


TO)| Tl TO)| Tl TO}| Tl TO}} Tl SMT_ID 
Core 0 Corel Core 0 Corel Core ID 
Package 0 Package 1 Package ID 


Figure 8-7. Topological Relationships between Hierarchical IDs in a Hypothetical MP Platform 


Table 8-2. Initial APIC IDs for the Logical Processors in a System that has Four Intel Xeon MP Processors Supporting 
Intel Hyper-Threading Technology! 


Initial APIC ID Package ID Core ID SMT ID 
OH OH OH OH 
1H OH OH 1H 
2H 1H OH OH 
3H 1H OH 1H 
4H 2H OH OH 
5H 2H OH 1H 
6H 3H OH OH 
7H 3H OH 1H 


NOTE: 


1. Because information on the number of processor cores in a physical package was not available in early single-core processors sup- 
porting Intel Hyper-Threading Technology, the core ID can be treated as 0. 


Table 8-3 shows the initial APIC IDs for a hypothetical situation with a dual processor system. Each physical 
package providing two processor cores, and each processor core also supporting Intel Hyper-Threading Tech- 
nology. 


Table 8-3. Initial APIC IDs for the Logical Processors in a System that has Two Physical Processors Supporting Dual- 
Core and Intel Hyper-Threading Technology 


Initial APIC ID Package ID Core ID SMT ID 
OH OH OH OH 
1H OH OH 1H 
2H OH 1H OH 
3H OH 1H 1H 
4H 1H OH OH 
5H 1H OH 1H 
6H 1H 1H OH 
7H 1H 1H 1H 
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8.9.3.1 Hierarchical ID of Logical Processors with x2APIC ID 


Table 8-4 shows an example of possible x2APIC ID assignments for a dual processor system that support x2APIC. 
Each physical package providing four processor cores, and each processor core also supporting Intel Hyper- 
Threading Technology. Note that the x2APIC ID need not be contiguous in the system. 


Table 8-4. Example of Possible x2APIC ID Assignment in a System that has Two Physical Processors Supporting 
x2APIC and Intel Hyper-Threading Technology 


x2APIC ID Package ID Core ID SMT ID 
OH OH OH OH 
1H OH OH 1H 
2H OH 1H OH 
3H OH 1H 1H 
4H OH 2H OH 
5H OH 2H 1H 
6H OH 3H OH 
7H OH 3H 1H 
10H 1H OH OH 
11H 1H OH 1H 
12H 1H 1H OH 
13H 1H 1H 1H 
14H 1H 2H OH 
15H 1H 2H 1H 
16H 1H 3H OH 
17H 1H 3H 1H 


8.9.4 Algorithm for Three-Level Mappings of APIC_ID 


Software can gather the initial APIC_IDs for each logical processor supported by the operating system at runtime? 
and extract identifiers corresponding to the three levels of sharing topology (package, core, and SMT). The three- 
level algorithms below focus on a non-clustered MP system for simplicity. They do not assume APIC IDs are contig- 
uous or that all logical processors on the platform are enabled. 


Intel supports multi-threading systems where all physical processors report identical values in CPUID leaf OBH, 
CPUID.1:EBX[23:16]), CPUID.4!°:EAX[31:26], and CPUID.41!:EAX[25:14]. The algorithms below assume the 
target system has symmetry across physical package boundaries with respect to the number of logical processors 
per package, number of cores per package, and cache topology within a package. 


Software can choose to assume three level hierarchy if it was developed to understand only three levels. However, 
software implementation needs to ensure it does not break if it runs on systems that have more levels in the hier- 
archy even if it does not recognize them. 


9. As noted in Section 8.6 and Section 8.9.3, the number of logical processors supported by the OS at runtime may be less than the 
total number logical processors available in the platform hardware. 


10. Maximum number of addressable ID for processor cores in a physical processor is obtained by executing CPUID with EAX=4 anda 
valid ECX index, The ECX index start at 0. 


11. Maximum number addressable ID for processor cores sharing the target cache level is obtained by executing CPUID with EAX = 4 
and the ECX index corresponding to the target cache level. 
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The extraction algorithm (for three-level mappings from an APIC ID) uses the general procedure depicted in 
Example 8-19, and is supplemented by more detailed descriptions on the derivation of topology enumeration 
parameters for extraction bit masks: 


1. Detect hardware multi-threading support in the processor. 


2. Derive a set of bit masks that can extract the sub ID of each hierarchical level of the topology. The algorithm to 
derive extraction bit masks for SMT_ID/CORE_ID/PACKAGE_ID differs based on APIC ID is 32-bit (see step 3 
below) or 8-bit (see step 4 below): 


3. If the processor supports CPUID leaf OBH, each APIC ID contains a 32-bit value, the topology enumeration 
parameters needed to derive three-level extraction bit masks are: 


a. 


fi 


Query the right-shift value for the SMT level of the topology using CPUID leaf OBH with ECX =OH as input. 
The number of bits to shift-right on x2APIC ID (EAX[4:0]) can distinguish different higher-level entities 
above SMT (e.g. processor cores) in the same physical package. This is also the width of the bit mask to 
extract the SMT_ID. 


Enumerate until the desired level is found (i.e. processor cores). Determine if the next level is the expected 
level. If the next level is not known to the software, keep enumerating until the next known or the last level. 
Software should use the previous level before this to represent the last previously known level (i.e. 
processor cores). If the software does not recognize or implement certain hierarchical levels, it should 
assume these unknown levels as an extension of the last known level. 


Query CPUID leaf OBH for the amount of bit shift to distinguish next higher-level entities (e.g. physical 
processor packages) in the system. This describes an explicit three-level-topology situation for commonly 
available processors. Consult Example 8-17 to adapt to situations beyond three-level topology of a physical 
processor. The width of the extraction bit mask can be used to derive the cumulative extraction bitmask to 
extract the sub IDs of logical processors (including different processor cores) in the same physical package. 
The extraction bit mask to distinguish merely different processor cores can be derived by xor’ing the SMT 
extraction bit mask from the cumulative extraction bit mask. 


Query the 32-bit x2APIC ID for the logical processor where the current thread is executing. 


Derive the extraction bit masks corresponding to SMT_ID, CORE_ID, and PACKAGE_ID, starting from 
SMT_ID. 


Apply each extraction bit mask to the 32-bit x2APIC ID to extract sub-field IDs. 


4. If the processor does not support CPUID leaf OBH, each initial APIC ID contains an 8-bit value, the topology 
enumeration parameters needed to derive extraction bit masks are: 


a. 


Query the size of address space for sub IDs that can accommodate logical processors in a physical 
processor package. This size parameters (CPUID.1:EBX[23:16]) can be used to derive the width of an 
extraction bitmask to enumerate the sub IDs of different logical processors in the same physical package. 


Query the size of address space for sub IDs that can accommodate processor cores in a physical processor 
package. This size parameters can be used to derive the width of an extraction bitmask to enumerate the 
sub IDs of processor cores in the same physical package. 


Query the 8-bit initial APIC ID for the logical processor where the current thread is executing. 


Derive the extraction bit masks using respective address sizes corresponding to SMT_ID, CORE_ID, and 
PACKAGE_ID, starting from SMT_ID. 


Apply each extraction bit mask to the 8-bit initial APIC ID to extract sub-field IDs. 
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Example 8-19. Support Routines for Detecting Hardware Multi-Threading and Identifying the Relationships Between Package, 
Core and Logical Processors 


1. | Detect support for Hardware Multi-Threading Support in a processor. 


// Returns a non-zero value if CPUID reports the presence of hardware multi-threading 
// support in the physical package where the current logical processor is located. 

// This does not guarantee BIOS or OS will enable all logical processors in the physical 
// package and make them available to applications. 

// Returns zero if hardware multi-threading is not present. 


#define HWMT_BIT 10000000H 


unsigned int HWMTSupported(void) 
{ 
// ensure cpuid instruction is supported 
execute cpuid with eax = 0 to get vendor string 
execute cpuid with eax = 1 to get feature flag and signature 


// Check to see if this a Genuine Intel Processor 


if (vendor string EQ Genuinelntel) { 
return (feature_flag_edx & HWMT_BIT); // bit 28 
} 


return 0; 


Example 8-20. Support Routines for Identifying Package, Core and Logical Processors from 32-bit x2APIC ID 


a. Derive the extraction bitmask for logical processors in a processor core and associated mask offset for different 
cores. 


int DeriveSMT_Mask_Offsets (void) 

{ 
if (HWMTSupported()) return -1; 
execute cpuid with eax = 11, ECX = 0; 
If (returned level type encoding in ECX[15:8] does not match SMT) return -1; 
Mask_SMT_shift = EAX[4:0]; // # bits shift right of APIC ID to distinguish different cores 
SMT_MASK = ~( (-1) << Mask_SMT_shift); // shift left to derive extraction bitmask for SMT_ID 
return O; 
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b. Derive the extraction bitmask for processor cores in a physical processor package and associated mask offset for 
different packages. 


int DeriveCore_Mask_Offsets (void) 


if (HWMTSupported()) return -1; 
execute cpuid with eax = 11, ECX = 0; 


while( ECX[1 5:8] ) { // level type encoding is valid 
Mask_Core_shift = EAX[4:0]; // needed to distinguish different physical packages 
ECX ++; 


execute cpuid with eax = 11; 


} 
COREPlusSMT_MASK = ~( (-1) << Mask_Core_shift); 


// treat levels between core and physical package as a core for software choosing not to implement or recognize 
// these unknown levels 

CORE_MASK = COREPlusSMT_MASK * SMT_MASK; 

PACKAGE_MASK = (-1) << Mask_Core_shift; 

return -1; 


c. Query the x2APIC ID of a logical processor. 


APIC_IDs for each logical processor. 


unsigned char Getx2APIC_ID (void) 


{ 
unsigned reg_edx = 0; 
execute cpuid with eax = 11,ECX = 0 
store returned value of edx 
return (unsigned) (reg_edx) ; 
} 


Example 8-21. Support Routines for Identifying Package, Core and Logical Processors from 8-bit Initial APIC ID 


a. _ Find the size of address space for logical processors in a physical processor package. 


#define NUM_LOGICAL_BITS OOFFOOOOH 
// Use the mask above and CPUID.1.EBX[23:16] to obtain the max number of addressable IDs 
// for logical processors in a physical package, 


//Returns the size of address space of logical processors in a physical processor package; 
// Software should not assume the value to be a power of 2. 


unsigned char MaxLPIDsPerPackage(void) 
{ 
if (HWMTSupported()) return 1; 
execute cpuid with eax = 1 
store returned value of ebx 
return (unsigned char) ((reg_ebx & NUM_LOGICAL_BITS) >> 16); 
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b. Find the size of address space for processor cores in a physical processor package. 


// Returns the max number of addressable IDs for processor cores in a physical processor package; 
// Software should not assume cpuid reports this value to be a power of 2. 


unsigned MaxCorelDsPerPackage(void) 
{ 
if (HWMTSupported()) return (unsigned char) 1; 
if cpuid supports leaf number 4 
{ // we can retrieve multi-core topology info using leaf 4 
execute cpuid with eax = 4, ecx = 0 
store returned value of eax 
return (unsigned) ((reg_eax >> 26) +1); 
} 
else // must be a single-core processor 
return 1; 


} 


c. Query the initial APIC ID of a logical processor. 


#define INITIAL_APIC_ID_BITS FFOOOOOOH // CPUID.1.€BX[31:24] initial APIC ID 
// Returns the 8-bit unique initial APIC ID for the processor running the code. 
// Software can use OS services to affinitize the current thread to each logical processor 


// available under the OS to gather the initial APIC_IDs for each logical processor. 


unsigned GetInitAPIC_ID (void) 


{ 

unsigned int reg_ebx = 0; 

execute cpuid with eax = 1 

store returned value of ebx 

return (unsigned) ((reg_ebx & INITIAL_APIC_ID_BITS) >> 24; 
} 
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d. Find the width of an extraction bitmask from the maximum count of the bit-field (address size). 


// Returns the mask bit width of a bit field from the maximum count that bit field can represent. 

// This algorithm does not assume ‘address size’ to have a value equal to power of 2. 

// Address size for SMT_ID can be calculated from MaxLPIDsPerPackage()/MaxCorelDsPerPackage() 
// Then use the routine below to derive the corresponding width of SMT extraction bitmask 

// Address size for CORE_ID is MaxCorelDsPerPackage(), 

// Derive the bitwidth for CORE extraction mask similarly 


unsigned FindMaskWidth(Unsigned Max_Count) 
{unsigned int mask_width, cnt = Max_Count; 
__asm { 
mov eax, cnt 
mov ecx, 0 
mov mask_width, ecx 
dec eax 
bDsr cx, ax 
jz next 
inc cx 
mov mask_width, ecx 
next: 
mov eax, mask_width 
} 
return mask_width; 


} 


e. Extract a sub ID from an 8-bit full ID, using address size of the sub ID and shift count. 


// The routine below can extract SMT_ID, CORE_ID, and PACKAGE_ID respectively from the init APIC_ID 

// To extract SMT_ID, MaxSublIDvalue is set to the address size of SMT_ID, Shift_Count = 0 

// To extract CORE_ID, MaxSubIDvalue is the address size of CORE_ID, Shift_Count is width of SMT extraction bitmask. 
// Returns the value of the sub ID, this is not a zero-based value 


Unsigned char GetSubID(unsigned char Full_ID, unsigned char MaxSub!IDvalue, unsigned char Shift_Count) 
{ 

MaskWidth = FindMaskWidth(MaxSubIDValue); 

MaskBits = ((uchar) (FFH << Shift_Count)) * ((uchar) (FFH << Shift_Count + MaskWidth)) ; 

SublD = Full_ID & MaskBits; 

Return SublD; 


Software must not assume local APIC_ID values in an MP system are consecutive. Non-consecutive local APIC_IDs 
may be the result of hardware configurations or debug features implemented in the BIOS or OS. 


An identifier for each hierarchical level can be extracted from an 8-bit APIC_ID using the support routines illus- 
trated in Example 8-21. The appropriate bit mask and shift value to construct the appropriate bit mask for each 
level must be determined dynamically at runtime. 


8.9.5 Identifying Topological Relationships in a MP System 
To detect the number of physical packages, processor cores, or other topological relationships in a MP system, the 
following procedures are recommended: 


¢ Extract the three-level identifiers from the APIC ID of each logical processor enabled by system software. The 
sequence is as follows (See the pseudo code shown in Example 8-22 and support routines shown in Example 
8-19): 
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The extraction start from the right-most bit field, corresponding to SMT_ID, the innermost hierarchy in 
a three-level topology (See Figure 8-7). For the right-most bit field, the shift value of the working mask 
is zero. The width of the bit field is determined dynamically using the maximum number of logical 
processor per core, which can be derived from information provided from CPUID. 


To extract the next bit-field, the shift value of the working mask is determined from the width of the bit 
mask of the previous step. The width of the bit field is determined dynamically using the maximum 
number of cores per package. 


To extract the remaining bit-field, the shift value of the working mask is determined from the maximum 
number of logical processor per package. So the remaining bits in the APIC ID (excluding those bits 
already extracted in the two previous steps) are extracted as the third identifier. This applies to a non- 
clustered MP system, or if there is no need to distinguish between PACKAGE_ID and CLUSTER_ID. 


If there is need to distinguish between PACKAGE_ID and CLUSTER_ID, PACKAGE_ID can be extracted 
using an algorithm similar to the extraction of CORE_ID, assuming the number of physical packages in 
each node of a clustered system is symmetric. 


e Assemble the three-level identifiers of SMT_ID, CORE_ID, PACKAGE_IDs into arrays for each enabled logical 
processor. This is shown in Example 8-23a. 


° To detect the number of physical packages: use PACKAGE_ID to identify those logical processors that reside in 
the same physical package. This is shown in Example 8-23b. This example also depicts a technique to construct 
a mask to represent the logical processors that reside in the same package. 


* To detect the number of processor cores: use CORE_ID to identify those logical processors that reside in the 
same core. This is shown in Example 8-23. This example also depicts a technique to construct a mask to 
represent the logical processors that reside in the same core. 


In Example 8-22, the numerical ID value can be obtained from the value extracted with the mask by shifting it right 
by shift count. Algorithms below do not shift the value. The assumption is that the SubID values can be compared 
for equivalence without the need to shift. 


Example 8-22. Pseudo Code Depicting Three-level Extraction Algorithm 


For Each local_APIC_ID{ 
// Calculate SMT_MASK, the bit mask pattern to extract SMT_ID, 
// SMT_MASK is determined using topology enumertaion parameters 
// from CPUID leaf OBH (Example 8-20); 
// otherwise, SMT_MASK is determined using CPUID leaf 01H and leaf 04H (Example 8-21). 
// This algorithm assumes there is symmetry across core boundary, i.e. each core within a 
// package has the same number of logical processors 
// SMT_ID always starts from bit 0, corresponding to the right-most bit-field 


SMT_ID = 


APIC_ID & SMT_MASK; 


// Extract CORE_ID: 
// CORE_MASK is determined in Example 8-20 or Example 8-21 


CORE_ID 


= (APIC_ID & CORE_MASK) ; 


// Extract PACKAGE_ID: 

// Assume single cluster. 

// Shift out the mask width for maximum logical processors per package 
// PACKAGE_MASK is determined in Example 8-20 or Example 8-21 
PACKAGE_ID = (APIC_ID & PACKAGE_MASK) ; 
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Example 8-23. Compute the Number of Packages, Cores, and Processor Relationships in a MP System 
a) Assemble lists of PACKAGE_ID, CORE_ID, and SMT_ID of each enabled logical processors 


//The BIOS and/or OS may limit the number of logical processors available to applications 
// after system boot. The below algorithm will compute topology for the processors visible 
// to the thread that is computing it. 


// Extract the 3-levels of IDs on every processor 

// SystemAffinity is a bitmask of all the processors started by the OS. Use OS specific APIs to 

// obtain it. 

// ThreadAffinityMask is used to affinitize the topology enumeration thread to each processor 
using OS specific APIs. 

// Allocate per processor arrays to store the Package_ID, Core_ID and SMT_ID for every started 
// processor. 


ThreadAffinityMask = 1; 
ProcessorNum = 0; 
while (ThreadAffinityMask + 0 && ThreadAffinityMask <= SystemAffinity) { 
// Check to make sure we can utilize this processor first. 
if (ThreadAffinityMask & SystemAffinity){ 
Set thread to run on the processor specified in ThreadAffinityMask 
Wait if necessary and ensure thread is running on specified processor 


APIC_ID = GetAPIC_ID(); // 32 bit ID in Example 8-20 or 8-bit ID in Example 8-21 
Extract the Package_ID, Core_ID and SMT_ID as explained in three level extraction 
algorithm of Example 8-22 
PackagelD[ProcessorNUM] = PACKAGE_ID; 
CorelD[ProcessorNum] = CORE_ID; 
SmtID[ProcessorNum] = SMT_ID; 
ProcessorNum++; 
} 
ThreadAffinityMask <<= 1; 


NumStartedLPs = ProcessorNum; 


b) Using the list of PACKAGE_ID to count the number of physical packages in a MP system and construct, for each package, a multi-bit 
mask corresponding to those logical processors residing in the same package. 


// Compute the number of packages by counting the number of processors 
// with unique PACKAGE_IDs in the PackagelD array. 
// Compute the mask of processors in each package. 


PackagelDBucket is an array of unique PACKAGE_ID values. Allocate an array of 
NumStartedLPs count of entries in this array. 

PackageProcessorMask is a corresponding array of the bit mask of processors belonging to 
the same package, these are processors with the same PACKAGE_ID 

The algorithm below assumes there is symmetry across package boundary if more than 
one socket is populated in an MP system. 

// Bucket Package IDs and compute processor mask for every package. 


PackageNum = 1; 

PackagelDBucket[O] = PackagelD[0]; 
ProcessorMask = 1; 
PackageProcessorMask[0] = ProcessorMask; 
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For (ProcessorNum = 1; ProcessorNum < NumStartedLPs; ProcessorNum++) { 
ProcessorMask << = 1; 
For (i=0; i < PackageNum; i++) { 
// we may be comparing bit-fields of logical processors residing in different 
// packages, the code below assume package symmetry 
If (PackagelD[ProcessorNum] = PackagelDBucket[i]) { 
PackageProcessorMask{[i] |= ProcessorMask; 
Break; // found in existing bucket, skip to next iteration 
} 
} 
if (i =PackageNum) { 
//PACKAGE_ID did not match any bucket, start new bucket 
PackagelDBucket[i] = PackagelD[ProcessorNum]; 
PackageProcessorMask[i] = ProcessorMask; 
PackageNum++; 
} 
} 
// PackageNum has the number of Packages started in OS 
// PackageProcessorMask[] array has the processor set of each package 


c) Using the list of CORE_ID to count the number of cores in a MP system and construct, for each core, a multi-bit mask corresponding 
to those logical processors residing in the same core. 


Processors in the same core can be determined by bucketing the processors with the same PACKAGE_ID and CORE_ID. Note that code 
below can BIT OR the values of PACKGE and CORE ID because they have not been shifted right. 


The algorithm below assumes there is symmetry across package boundary if more than one socket is populated in an MP system. 


//Bucketing PACKAGE and CORE IDs and computing processor mask for every core 
CoreNum = 1; 
CorelDBucket[0] = PackagelD[0] | CorelD[O}]; 
ProcessorMask = 1; 
CoreProcessorMask[0] = ProcessorMask; 
For (ProcessorNum = 1; ProcessorNum < NumStartedLPs; ProcessorNum++) { 
ProcessorMask << = 1; 
For (i=0; i < CoreNum; i++) { 
// we may be comparing bit-fields of logical processors residing in different 
// packages, the code below assume package symmetry 
If ((PackagelD[ProcessorNum] | CorelD[ProcessorNum]) = CorelDBucket{i]) { 
CoreProcessorMask[i] |= ProcessorMask; 
Break; // found in existing bucket, skip to next iteration 
} 
} 
if (i = CoreNum) { 
//Did not match any bucket, start new bucket 
CorelDBucket[i] = PackagelD[ProcessorNum] | CorelD[ProcessorNum]; 
CoreProcessorMask[i] = ProcessorMask; 
CoreNum++; 
} 
} 
// CoreNum has the number of cores started in the OS 
// CoreProcessorMask[] array has the processor set of each core 


Other processor relationships such as processor mask of sibling cores can be computed from set operations of the 
PackageProcessorMask[] and CoreProcessorMask[]. 
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The algorithm shown above can be adapted to work with earlier generations of single-core IA-32 processors that 
support Intel Hyper-Threading Technology and in situations that the deterministic cache parameter leaf is not 
supported (provided CPUID supports initial APIC ID). A reference code example is available (see Inte/® 64 Archi- 
tecture Processor Topology Enumeration). 


8.10 MANAGEMENT OF IDLE AND BLOCKED CONDITIONS 


When a logical processor in an MP system (including multi-core processor or processors supporting Intel Hyper- 
Threading Technology) is idle (no work to do) or blocked (on a lock or semaphore), additional management of the 
core execution engine resource can be accomplished by using the HLT (halt), PAUSE, or the MONITOR/MWAIT 
instructions. 


8.10.1 HLT Instruction 


The HLT instruction stops the execution of the logical processor on which it is executed and places it in a halted 
state until further notice (see the description of the HLT instruction in Chapter 3 of the Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 2A). When a logical processor is halted, active logical processors 
continue to have full access to the shared resources within the physical package. Here shared resources that were 
being used by the halted logical processor become available to active logical processors, allowing them to execute 
at greater efficiency. When the halted logical processor resumes execution, shared resources are again shared 
among all active logical processors. (See Section 8.10.6.3, “Halt Idle Logical Processors,” for more information 
about using the HLT instruction with processors supporting Intel Hyper-Threading Technology.) 


8.10.2 PAUSE Instruction 


The PAUSE instruction can improves the performance of processors supporting Intel Hyper-Threading Technology 
when executing “spin-wait loops” and other routines where one thread is accessing a shared lock or semaphore in 
a tight polling loop. When executing a spin-wait loop, the processor can suffer a severe performance penalty when 
exiting the loop because it detects a possible memory order violation and flushes the core processor's pipeline. The 
PAUSE instruction provides a hint to the processor that the code sequence is a spin-wait loop. The processor uses 
this hint to avoid the memory order violation and prevent the pipeline flush. In addition, the PAUSE instruction de- 
pipelines the spin-wait loop to prevent it from consuming execution resources excessively and consume power 
needlessly. (See Section 8.10.6.1, “Use the PAUSE Instruction in Spin-Wait Loops,” for more information about 
using the PAUSE instruction with IA-32 processors supporting Intel Hyper-Threading Technology.) 


8.10.3. Detecting Support MONITOR/MWAIT Instruction 


Streaming SIMD Extensions 3 introduced two instructions (MONITOR and MWAIT) to help multithreaded software 
improve thread synchronization. In the initial implementation, MONITOR and MWAIT are available to software at 
ring O. The instructions are conditionally available at levels greater than 0. Use the following steps to detect the 
availability of MONITOR and MWAIT: 


¢ Use CPUID to query the MONITOR bit (CPUID.1.ECX[3] = 1). 


¢ If CPUID indicates support, execute MONITOR inside a TRY/EXCEPT exception handler and trap for an 
exception. If an exception occurs, MONITOR and MWAIT are not supported at a privilege level greater than 0. 
See Example 8-24. 
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Example 8-24. Verifying MONITOR/MWAIT Support 


boolean MONITOR_MWAIT_works = TRUE; 
try { 
_asm { 
XOFr @CX, @CX 
xor edx, edx 
mov eax, MemArea 
monitor 
} 
// Use monitor 
} except (UNWIND) { 
// if we get here, MONITOR/MWAIT is not supported 
MONITOR_MWAIT_works = FALSE; 


8.10.4 MONITOR/MWAIT Instruction 


Operating systems usually implement idle loops to handle thread synchronization. In a typical idle-loop scenario, 

there could be several “busy loops” and they would use a set of memory locations. An impacted processor waits in 
a loop and poll a memory location to determine if there is available work to execute. The posting of work is typically 
a write to memory (the work-queue of the waiting processor). The time for initiating a work request and getting it 
scheduled is on the order of a few bus cycles. 


From a resource sharing perspective (logical processors sharing execution resources), use of the HLT instruction in 
an OS idle loop is desirable but has implications. Executing the HLT instruction on a idle logical processor puts the 
targeted processor in a non-execution state. This requires another processor (when posting work for the halted 
logical processor) to wake up the halted processor using an inter-processor interrupt. The posting and servicing of 
such an interrupt introduces a delay in the servicing of new work requests. 


In a shared memory configuration, exits from busy loops usually occur because of a state change applicable to a 
specific memory location; such a change tends to be triggered by writes to the memory location by another agent 
(typically a processor). 


MONITOR/MWAIT complement the use of HLT and PAUSE to allow for efficient partitioning and un-partitioning of 
shared resources among logical processors sharing physical resources. MONITOR sets up an effective address 
range that is monitored for write-to-memory activities; MWAIT places the processor in an optimized state (this may 
vary between different implementations) until a write to the monitored address range occurs. 


In the initial implementation of MONITOR and MWAIT, they are available at CPL = 0 only. 


Both instructions rely on the state of the processor’s monitor hardware. The monitor hardware can be either armed 
(by executing the MONITOR instruction) or triggered (due to a variety of events, including a store to the monitored 
memory region). If upon execution of MWAIT, monitor hardware is in a triggered state: MWAIT behaves as a NOP 
and execution continues at the next instruction in the execution stream. The state of monitor hardware is not archi- 
tecturally visible except through the behavior of MWAIT. 


Multiple events other than a write to the triggering address range can cause a processor that executed MWAIT to 
wake up. These include events that would lead to voluntary or involuntary context switches, such as: 


¢  ~=External interrupts, including NMI, SMI, INIT, BINIT, MCERR, A2OM# 
¢ Faults, Aborts (including Machine Check) 


e¢ = Architectural TLB invalidations including writes to CRO, CR3, CR4 and certain MSR writes; execution of LMSW 
(occurring prior to issuing MWAIT but after setting the monitor) 


¢ Voluntary transitions due to fast system call and far calls (occurring prior to issuing MWAIT but after setting the 
monitor) 


Power management related events (such as Thermal Monitor 2 or chipset driven STPCLK# assertion) will not cause 
the monitor event pending flag to be cleared. Faults will not cause the monitor event pending flag to be cleared. 
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Software should not allow for voluntary context switches in between MONITOR/MWAIT in the instruction flow. Note 
that execution of MWAIT does not re-arm the monitor hardware. This means that MONITOR/MWAIT need to be 
executed in a loop. Also note that exits from the MWAIT state could be due to a condition other than a write to the 
triggering address; software should explicitly check the triggering data location to determine if the write occurred. 
Software should also check the value of the triggering address following the execution of the monitor instruction 
(and prior to the execution of the MWAIT instruction). This check is to identify any writes to the triggering address 
that occurred during the course of MONITOR execution. 


The address range provided to the MONITOR instruction must be of write-back caching type. Only write-back 
memory type stores to the monitored address range will trigger the monitor hardware. If the address range is not 
in memory of write-back type, the address monitor hardware may not be set up properly or the monitor hardware 
may not be armed. Software is also responsible for ensuring that 


¢ Writes that are not intended to cause the exit of a busy loop do not write to a location within the address region 
being monitored by the monitor hardware, 


¢ Writes intended to cause the exit of a busy loop are written to locations within the monitored address region. 


Not doing so will lead to more false wakeups (an exit from the MWAIT state not due to a write to the intended data 
location). These have negative performance implications. It might be necessary for software to use padding to 
prevent false wakeups. CPUID provides a mechanism for determining the size data locations for monitoring as well 
as a mechanism for determining the size of a the pad. 


8.10.5 Monitor/Mwait Address Range Determination 


To use the MONITOR/MWAIT instructions, software should know the length of the region monitored by the 
MONITOR/MWAIT instructions and the size of the coherence line size for cache-snoop traffic in a multiprocessor 
system. This information can be queried using the CPUID monitor leaf function (EAX = 05H). You will need the 
smallest and largest monitor line size: 


* To avoid missed wake-ups: make sure that the data structure used to monitor writes fits within the smallest 
monitor line-size. Otherwise, the processor may not wake up after a write intended to trigger an exit from 
MWAITT. 


* To avoid false wake-ups; use the largest monitor line size to pad the data structure used to monitor writes. 
Software must make sure that beyond the data structure, no unrelated data variable exists in the triggering 
area for MWAIT. A pad may be needed to avoid this situation. 


These above two values bear no relationship to cache line size in the system and software should not make any 
assumptions to that effect. Within a single-cluster system, the two parameters should default to be the same (the 
size of the monitor triggering area is the same as the system coherence line size). 


Based on the monitor line sizes returned by the CPUID, the OS should dynamically allocate structures with appro- 
priate padding. If static data structures must be used by an OS, attempt to adapt the data structure and use a 
dynamically allocated data buffer for thread synchronization. When the latter technique is not possible, consider 
not using MONITOR/MWAIT when using static data structures. 


To set up the data structure correctly for MONITOR/MWAIT on multi-clustered systems: interaction between 
processors, chipsets, and the BIOS is required (system coherence line size may depend on the chipset used in the 
system; the size could be different from the processor’s monitor triggering area). The BIOS is responsible to set the 
correct value for system coherence line size using the IA32_MONITOR_FILTER_LINE_SIZE MSR. Depending on the 
relative magnitude of the size of the monitor triggering area versus the value written into the 
IA32_MONITOR_FILTER_LINE_SIZE MSR, the smaller of the parameters will be reported as the Smallest Monitor 
Line Size. The larger of the parameters will be reported as the Largest Monitor Line Size. 


8.10.6 Required Operating System Support 


This section describes changes that must be made to an operating system to run on processors supporting Intel 
Hyper-Threading Technology. It also describes optimizations that can help an operating system make more effi- 
cient use of the logical processors sharing execution resources. The required changes and suggested optimizations 
are representative of the types of modifications that appear in Windows* XP and Linux* kernel 2.4.0 operating 
systems for Intel processors supporting Intel Hyper-Threading Technology. Additional optimizations for processors 
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supporting Intel Hyper-Threading Technology are described in the Inte/® 64 and IA-32 Architectures Optimization 
Reference Manual. 


8.10.6.1 Use the PAUSE Instruction in Spin-Wait Loops 


Intel recommends that a PAUSE instruction be placed in all spin-wait loops that run on Intel processors supporting 
Intel Hyper-Threading Technology and multi-core processors. 


Software routines that use spin-wait loops include multiprocessor synchronization primitives (spin-locks, sema- 
phores, and mutex variables) and idle loops. Such routines keep the processor core busy executing a load-compare- 
branch loop while a thread waits for a resource to become available. Including a PAUSE instruction in such a loop 
greatly improves efficiency (see Section 8.10.2, “PAUSE Instruction”). The following routine gives an example of a 
spin-wait loop that uses a PAUSE instruction: 


Spin_Lock: 
CMP lockvar, O ;Check if lock is free 
JE Get_Lock 
PAUSE Short delay 
JMP Spin_Lock 
Get_Lock: 
MOV EAX, 1 
XCHG EAX, lockvar ;Try to get lock 
CMP EAX, 0 Test if successful 
JNE Spin_Lock 
Critical_Section: 
<critical section code> 
MOV lockvar, 0 


Continue: 


The spin-wait loop above uses a “test, test-and-set” technique for determining the availability of the synchroniza- 
tion variable. This technique is recommended when writing spin-wait loops. 


In IA-32 processor generations earlier than the Pentium 4 processor, the PAUSE instruction is treated as a NOP 
instruction. 


8.10.6.2 Potential Usage of MONITOR/MWAIT in CO Idle Loops 


An operating system may implement different handlers for different idle states. A typical OS idle loop on an ACPI- 
compatible OS is shown in Example 8-25: 


Example 8-25. A Typical OS Idle Loop 


// WorkQueue is a memory location indicating there is a thread 

// ready to run. A non-zero value for WorkQueue is assumed to 

// indicate the presence of work to be scheduled on the processor. 
// The idle loop is entered with interrupts disabled. 


WHILE (1) { 

IF (WorkQueue) THEN { 
// Schedule work at WorkQueue. 
} 

ELSE { 
// No work to do - wait in appropriate C-state handler depending 
// on Idle time accumulated 
IF (IdleTime >= IdleTimeThreshhold) THEN { 

// Call appropriate C1, C2, C3 state handler, C1 handler 
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// shown below 
} 
} 
} 
// C1 handler uses a Halt instruction 
VOID C1Handler() 
{ STI 
HLT 


} 


The MONITOR and MWAIT instructions may be considered for use in the CO idle state loops, if MONITOR and MWAIT are supported. 


Example 8-26. An OS Idle Loop with MONITOR/MWAIT in the CO Idle Loop 


// \WorkQueue is a memory location indicating there is a thread 

// ready to run. A non-zero value for WorkQueue is assumed to 

// indicate the presence of work to be scheduled on the processor. 

// The following example assumes that the necessary padding has been 
// added surrounding WorkQueue to eliminate false wakeups 

// The idle loop is entered with interrupts disabled. 


WHILE (1) { 
IF (WorkQueue) THEN { 
// Schedule work at WorkQueue. 


} 
ELSE { 
// No work to do - wait in appropriate C-state handler depending 
// on Idle time accumulated. 
IF (IdleTime >= IdleTimeThreshhold) THEN { 
// Call appropriate C1, C2, C3 state handler, C1 
// handler shown below 
MONITOR WorkQueue // Setup of eax with WorkQueue 


// LinearAddress, 
// ECX, EDX = 0 
IF (WorkQueue = 0) THEN { 
MWAIT 
} 
} 
} 
} 
// C1 handler uses a Halt instruction. 
VOID C1Handler() 
{ STI 
HLT 
} 


8.10.6.3 Halt Idle Logical Processors 


If one of two logical processors is idle or in a spin-wait loop of long duration, explicitly halt that processor by means 
of a HLT instruction. 


In an MP system, operating systems can place idle processors into a loop that continuously checks the run queue 
for runnable software tasks. Logical processors that execute idle loops consume a significant amount of core’s 
execution resources that might otherwise be used by the other logical processors in the physical package. For this 
reason, halting idle logical processors optimizes the performance. ?? If all logical processors within a physical 
package are halted, the processor will enter a power-saving state. 
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8.10.6.4 Potential Usage of MONITOR/MWAIT in C1 Idle Loops 


An operating system may also consider replacing HLT with MONITOR/MWAIT in its C1 idle loop. An example is 
shown in Example 8-27: 


Example 8-27. An OS Idle Loop with MONITOR/MWAIT in the C1 Idle Loop 


// WorkQueue is a memory location indicating there is a thread 

// ready to run. A non-zero value for WorkQueue is assumed to 

// indicate the presence of work to be scheduled on the processor. 

// The following example assumes that the necessary padding has been 
// added surrounding WorkQueue to eliminate false wakeups 

// The idle loop is entered with interrupts disabled. 


WHILE (1) { 

IF (WorkQueue) THEN { 
// Schedule work at WorkQueue 
} 

ELSE { 
// No work to do - wait in appropriate C-state handler depending 
// on Idle time accumulated 
IF (IdleTime >= IdleTimeThreshhold) THEN { 
// Call appropriate C1, C2, C3 state handler, C1 
// handler shown below 


} 
} 


VOID C1Handler() 


{ MONITOR WorkQueue // Setup of eax with WorkQueue LinearAddress, 


// ECX, EDX = O 
IF (WorkQueue = 0) THEN { 
STI 
MWAIT // EAX, ECX = 0 
} 


8.10.6.5 Guidelines for Scheduling Threads on Logical Processors Sharing Execution Resources 


Because the logical processors, the order in which threads are dispatched to logical processors for execution can 
affect the overall efficiency of a system. The following guidelines are recommended for scheduling threads for 
execution. 


* Dispatch threads to one logical processor per processor core before dispatching threads to the other logical 
processor sharing execution resources in the same processor core. 


* In an MP system with two or more physical packages, distribute threads out over all the physical processors, 
rather than concentrate them in one or two physical processors. 


e Use processor affinity to assign a thread to a specific processor core or package, depending on the cache- 
sharing topology. The practice increases the chance that the processor’s caches will contain some of the 
thread’s code and data when it is dispatched for execution after being suspended. 


12. Excessive transitions into and out of the HALT state could also incur performance penalties. Operating systems should evaluate the 
performance trade-offs for their operating system. 
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8.10.6.6 Eliminate Execution-Based Timing Loops 


Intel discourages the use of timing loops that depend on a processor's execution speed to measure time. There are 
several reasons: 


¢ Timing loops cause problems when they are calibrated on a IA-32 processor running at one frequency and then 
executed on a processor running at another frequency. 


¢ Routines for calibrating execution-based timing loops produce unpredictable results when run on an IA-32 
processor supporting Intel Hyper-Threading Technology. This is due to the sharing of execution resources 
between the logical processors within a physical package. 


To avoid the problems described, timing loop routines must use a timing mechanism for the loop that does not 
depend on the execution speed of the logical processors in the system. The following sources are generally avail- 
able: 


¢ Ahigh resolution system timer (for example, an Intel 8254). 
¢ Ahigh resolution timer within the processor (such as, the local APIC timer or the time-stamp counter). 
For additional information, see the Inte/® 64 and IA-32 Architectures Optimization Reference Manual. 


8.10.6.7 Place Locks and Semaphores in Aligned, 128-Byte Blocks of Memory 


When software uses locks or semaphores to synchronize processes, threads, or other code sections; Intel recom- 
mends that only one lock or semaphore be present within a cache line (or 128 byte sector, if 128-byte sector is 
supported). In processors based on Intel NetBurst microarchitecture (which support 128-byte sector consisting of 
two cache lines), following this recommendation means that each lock or semaphore should be contained in a 128- 
byte block of memory that begins on a 128-byte boundary. The practice minimizes the bus traffic required to 
service locks. 


8.11 MP INITIALIZATION FOR P6 FAMILY PROCESSORS 


This section describes the MP initialization process for systems that use multiple P6 family processors. This process 
uses the MP initialization protocol that was introduced with the Pentium Pro processor (see Section 8.4, “Multiple- 
Processor (MP) Initialization”). For P6 family processors, this protocol is typically used to boot 2 or 4 processors 
that reside on single system bus; however, it can support from 2 to 15 processors in a multi-clustered system when 
the APIC busses are tied together. Larger systems are not supported. 


8.11.1 Overview of the MP Initialization Process For P6 Family Processors 


During the execution of the MP initialization protocol, one processor is selected as the bootstrap processor (BSP) 
and the remaining processors are designated as application processors (APs), see Section 8.4.1, “BSP and AP 
Processors.” Thereafter, the BSP manages the initialization of itself and the APs. This initialization includes 
executing BIOS initialization code and operating-system initialization code. 


The MP protocol imposes the following requirements and restrictions on the system: 
¢ An APIC clock (APICLK) must be provided. 


¢ The MP protocol will be executed only after a power-up or RESET. If the MP protocol has been completed and a 
BSP has been chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP 
protocol to be repeated. Instead, each processor examines its BSP flag (in the APIC_BASE MSR) to determine 
whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an 
AP). 


¢ All devices in the system that are capable of delivering interrupts to the processors must be inhibited from 
doing so for the duration of the MP initialization protocol. The time during which interrupts must be inhibited 
includes the window between when the BSP issues an INIT-SIPI-SIPI sequence to an AP and when the AP 
responds to the last SIPI in the sequence. 


The following special-purpose interprocessor interrupts (IPIs) are used during the boot phase of the MP initializa- 
tion protocol. These IPIs are broadcast on the APIC bus. 
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* Boot IPI (BIPI)—Initiates the arbitration mechanism that selects a BSP from the group of processors on the 
system bus and designates the remainder of the processors as APs. Each processor on the system bus 
broadcasts a BIPI to all the processors following a power-up or RESET. 


¢ Final Boot IPI (FIPI)—Initiates the BIOS initialization procedure for the BSP. This IPI is broadcast to all the 
processors on the system bus, but only the BSP responds to it. The BSP responds by beginning execution of the 
BIOS initialization code at the reset vector. 


¢ Startup IPI (SIPI)—Initiates the initialization procedure for an AP. The SIPI message contains a vector to the AP 
initialization code in the BIOS. 


Table 8-5 describes the various fields of the boot phase IPIs. 


Table 8-5. Boot Phase IPI Message Format 


Type Destination Destination Trigger Destination Delivery Vector 
yp Field Shorthand Mode Level Mode Mode (Hex) 

BIPI Not used Allincluding self | Edge Deassert Don't Care Fixed 40 to 4E* 
(000) 

FIPI Not used Allincluding self | Edge Deassert Don't Care Fixed 10 
(000) 

SIPI Used All excluding self | Edge Assert Physical StartUp 00 to FF 
(110) 


NOTE: 
* For all P6 family processors. 


For BIPI messages, the lower 4 bits of the vector field contain the APIC ID of the processor issuing the message and 
the upper 4 bits contain the “generation ID” of the message. All P6 family processor will have a generation ID of 4H. 
BIPIs will therefore use vector values ranging from 40H to 4EH (4FH can not be used because FH is not a valid APIC 
ID). 


8.11.2 MP Initialization Protocol Algorithm 


Following a power-up or RESET of a system, the P6 family processors in the system execute the MP initialization 
protocol algorithm to initialize each of the processors on the system bus. In the course of executing this algorithm, 
the following boot-up and initialization operations are carried out: 


1. Each processor on the system bus is assigned a unique APIC ID, based on system topology (see Section 8.4.5, 
“Identifying Logical Processors in an MP System”). This ID is written into the local APIC ID register for each 
processor. 


2. Each processor executes its internal BIST simultaneously with the other processors on the system bus. Upon 
completion of the BIST (at TO), each processor broadcasts a BIPI to “all including self” (see Figure 8-1). 


3. APIC arbitration hardware causes all the APICs to respond to the BIPIs one at a time (at T1, T2, T3, and T4). 


When the first BIPI is received (at time T1), each APIC compares the four least significant bits of the BIPI’s 
vector field with its APIC ID. If the vector and APIC ID match, the processor selects itself as the BSP by setting 
the BSP flag in its IA32_APIC_BASE MSR. If the vector and APIC ID do not match, the processor selects itself 
as an AP by entering the “wait for SIPI” state. (Note that in Figure 8-1, the BIPI from processor 1 is the first 
BIPI to be handled, so processor 1 becomes the BSP.) 


5. The newly established BSP broadcasts an FIPI message to “all including self.” The FIPI is guaranteed to be 
handled only after the completion of the BIPIs that were issued by the non-BSP processors. 


8-56 Vol. 3A 


10. 


11. 


12. 


MULTIPLE-PROCESSOR MANAGEMENT 


System (CPU) Bus 


ek a a a aad 


Pentium III Pentium III Pentium III Pentium III 
Processor 0 Processor 1 Processor 2 Processor 3 
APIC Bus 


Processor 1 

Becomes BSP 
TO T1 T2 T3 T4 T5 
A A A A A A 


BIPI.1 BIPI.0 BIPI.3 BIPI.2 FIPI 
Serial Bus Activity 


Figure 8-1. MP System With Multiple Pentium III Processors 


After the BSP has been established, the outstanding BIPIs are received one at a time (at T2, T3, and T4) and 
ignored by all processors. 


When the FIPI is finally received (at T5), only the BSP responds to it. It responds by fetching and executing 
BIOS boot-strap code, beginning at the reset vector (physical address FFFF FFFOH). 


As part of the boot-strap code, the BSP creates an ACPI table and an MP table and adds its initial APIC ID to 
these tables as appropriate. 


At the end of the boot-strap procedure, the BSP broadcasts a SIPI message to all the APs in the system. Here, 
the SIPI message contains a vector to the BIOS AP initialization code (at OOOV VOOOH, where VV is the vector 
contained in the SIPI message). 


All APs respond to the SIPI message by racing to a BIOS initialization semaphore. The first one to the 
semaphore begins executing the initialization code. (See MP init code for semaphore implementation details. ) 
As part of the AP initialization procedure, the AP adds its APIC ID number to the ACPI and MP tables as appro- 
priate. At the completion of the initialization procedure, the AP executes a CLI instruction (to clear the IF flag in 
the EFLAGS register) and halts itself. 


When each of the APs has gained access to the semaphore and executed the AP initialization code and all 
written their APIC IDs into the appropriate places in the ACPI and MP tables, the BSP establishes a count for the 
number of processors connected to the system bus, completes executing the BIOS boot-strap code, and then 
begins executing operating-system boot-strap and start-up code. 


While the BSP is executing operating-system boot-strap and start-up code, the APs remain in the halted state. 
In this state they will respond only to INITs, NMIs, and SMIs. They will also respond to snoops and to assertions 
of the STPCLK# pin. 


See Section 8.4.4, “MP Initialization Example,” for an annotated example the use of the MP protocol to boot IA-32 
processors in an MP. This code should run on any IA-32 processor that used the MP protocol. 
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8.11.2.1. Error Detection and Handling During the MP Initialization Protocol 


Errors may occur on the APIC bus during the MP initialization phase. These errors may be transient or permanent 
and can be caused by a variety of failure mechanisms (for example, broken traces, soft errors during bus usage, 
etc.). All serial bus related errors will result in an APIC checksum or acceptance error. 


The MP initialization protocol makes the following assumptions regarding errors that occur during initialization: 


e If errors are detected on the APIC bus during execution of the MP initialization protocol, the processors that 
detect the errors are shut down. 


¢ The MP initialization protocol will be executed by processors even if they fail their BIST sequences. 
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CHAPTER 9 
PROCESSOR MANAGEMENT AND INITIALIZATION 


This chapter describes the facilities provided for managing processor wide functions and for initializing the 
processor. The subjects covered include: processor initialization, x87 FPU initialization, processor configuration, 
feature determination, mode switching, the MSRs (in the Pentium, P6 family, Pentium 4, and Intel Xeon proces- 
sors), and the MTRRs (in the P6 family, Pentium 4, and Intel Xeon processors). 


9.1 INITIALIZATION OVERVIEW 


Following power-up or an assertion of the RESET# pin, each processor on the system bus performs a hardware 
initialization of the processor (known as a hardware reset) and an optional built-in self-test (BIST). A hardware 
reset sets each processor’s registers to a known state and places the processor in real-address mode. It also inval- 
idates the internal caches, translation lookaside buffers (TLBs) and the branch target buffer (BTB). At this point, 
the action taken depends on the processor family: 


¢ Pentium 4 processors (CPUID DisplayFamily OFH) — All the processors on the system bus (including a 
single processor in a uniprocessor system) execute the multiple processor (MP) initialization protocol. The 
processor that is selected through this protocol as the bootstrap processor (BSP) then immediately starts 
executing software-initialization code in the current code segment beginning at the offset in the EIP register. 
The application (non-BSP) processors (APs) go into a Wait For Startup IPI (SIPI) state while the BSP is 
executing initialization code. See Section 8.4, “Multiple-Processor (MP) Initialization,” for more details. Note 
that in a uniprocessor system, the single Pentium 4 or Intel Xeon processor automatically becomes the BSP. 


¢ IA-32 and Intel 64 processors (CPUID DisplayFamily 06H) — The action taken is the same as for the 
Pentium 4 processors (as described in the previous paragraph). 


¢ Pentium processors — In either a single- or dual- processor system, a single Pentium processor is always 
pre-designated as the primary processor. Following a reset, the primary processor behaves as follows in both 
single- and dual-processor systems. Using the dual-processor (DP) ready initialization protocol, the primary 
processor immediately starts executing software-initialization code in the current code segment beginning at 
the offset in the EIP register. The secondary processor (if there is one) goes into a halt state. 


° Intel486 processor — The primary processor (or single processor in a uniprocessor system) immediately 
starts executing software-initialization code in the current code segment beginning at the offset in the EIP 
register. (The Intel486 does not automatically execute a DP or MP initialization protocol to determine which 
processor is the primary processor.) 


The software-initialization code performs all system-specific initialization of the BSP or primary processor and the 
system logic. 


At this point, for MP (or DP) systems, the BSP (or primary) processor wakes up each AP (or secondary) processor 
to enable those processors to execute self-configuration code. 


When all processors are initialized, configured, and synchronized, the BSP or primary processor begins executing 
an initial operating-system or executive task. 


The x87 FPU is also initialized to a known state during hardware reset. x87 FPU software initialization code can then 
be executed to perform operations such as setting the precision of the x87 FPU and the exception masks. No special 
initialization of the x87 FPU is required to switch operating modes. 


Asserting the INIT# pin on the processor invokes a similar response to a hardware reset. The major difference is 
that during an INIT, the internal caches, MSRs, MTRRs, and x87 FPU state are left unchanged (although, the TLBs 
and BTB are invalidated as with a hardware reset). An INIT provides a method for switching from protected to real- 
address mode while maintaining the contents of the internal caches. 
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9.1.1 


Processor State After Reset 


Following power-up, The state of control register CRO is 6(0000010H (see Figure 9-1). This places the processor is 
in real-address mode with paging disabled. 


Paging disabled: 0 
Caching disabled: 1 


Not write-through disabled: 1 


Alignment check disabled: 0 
Write-protect disabled: 0 


31 30 29 28 1918 171615 6543210 
P|C|N Al |W N},|T/E|M|P 
enn Reserved ML |p Reserved El'lsimipie 


External x87 FPU error reporting: 0 


(Not used): 1 


No task switch: 0 


x87 FPU instructions not trapped: 0 
WAIT/FWAIT instructions not trapped: 0 


Real-address mode: 0 


Figure 9-1. Contents of CRO Register after Reset 


The state of the flags and other registers following power-up for the Pentium 4, Pentium Pro, and Pentium proces- 
sors are shown in Section 21.39, “Initial State of Pentium, Pentium Pro and Pentium 4 Processors” of the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 3B. 


Table 9-1 shows processor states of IA-32 and Intel 64 processors with CPUID DisplayFamily signature of O6H at 
the following events: power-up, RESET, and INIT. In a few cases, the behavior of some registers behave slightly 
different across warm RESET, the variant cases are marked in Table 9-1 and described in more detail in Table 9-2. 


Table 9-1. IA-32 and Intel 64 Processor States Following Power-up, Reset, or INIT 


Base = FFFFOOOOH 
Limit = FFFFH 
AR = Present, R/W, Accessed 


Base = FFFFOOOOH 
Limit = FFFFH 
AR = Present, R/W, Accessed 


Register Power up Reset INIT 
EFLAGS' 00000002H 00000002H 00000002H 
EIP OOOOFFFOH OOOOFFFOH OOOOFFFOH 
CRO 60000010H2 6000001 0H? 60000010H? 
CR2, CR3, CR4 O00000000H O0000000H O0000000H 
cs Selector = FOOOH Selector = FOOOH Selector = FOOOH 


Base = FFFFOOOOH 
Limit = FFFFH 
AR = Present, R/W, Accessed 


SS, DS, ES, FS, GS 


Selector = OOOOH 
Base = OOO000000H 


Selector = OOOOH 
Base = OO000000H 


Selector = OOOOH 
Base = OOO000000H 


Limit = FFFFH Limit = FFFFH Limit = FFFFH 
AR = Present, R/W, Accessed AR = Present, R/W, Accessed AR = Present, R/W, Accessed 
EDX 000n06xxH? O00n06xxH? O00n06xxH? 
EAX 0 04 0 
EBX, ECX, ESI, EDI, EBP, ESP | OOOOO000H O0000000H OO000000H 
STO through ST7° +0.0 +0.0 FINIT/FNINIT: Unchanged 
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Table 9-1. IA-32 and Intel 64 Processor States Following Power-up, Reset, or INIT (Contd.) 


Register Power up Reset INIT 
x87 FPU Control Word? 0040H 0040H FINIT/FNINIT: 037FH 
x87 FPU Status Word® O000H 0O00H FINIT/FNINIT: OOOOH 
x87 FPU Tag Word? 5555H 5555H FINIT/FNINIT: FFFFH 
x87 FPU Data Operand and 0000H OO00H FINIT/FNINIT: OOOOH 
CS Seg. Selectors® 
x87 FPU Data Operand and OO000000H OQO000000H FINIT/FNINIT: OOOOOOOOH 
Inst. Pointers> 
MMO through MM7> OQOO0000000000000H OQOO0000000000000H INIT or FINIT/FNINIT: Unchanged 
XMMO through XMM7 OH OH Unchanged 
MXCSR 1F80H 1F80H Unchanged 
GDTR, IDTR Base = QO000000H Base = OOO00000H Base = OQOO0O0000H 

Limit = FFFFH Limit = FFFFH Limit = FFFFH 


AR = Present, R/W 


AR = Present, R/W 


AR = Present, R/W 


LDTR, Task Register 


Selector = OOOOH 
Base = OOO000000H 


Selector = OOOOH 
Base = OOO000000H 


Selector = OOOOH 
Base = OOO00000H 


Limit = FFFFH Limit = FFFFH Limit = FFFFH 

AR = Present, R/W AR = Present, R/W AR = Present, R/W 
DRO, DR1, DR2, DR3 O0000000H O00000000H O0000000H 
DR6 FFFFOFFOH FFFFOFFOH FFFFOFFOH 
DR7 00000400H 00000400H 00000400H 
R8-R15 O000000000000000H O000000000000000H O000000000000000H 
XMM8-XMM15 OH OH Unchanged 
XCRO 1H 1H Unchanged 
IA32_XSS OH OH Unchanged 
YMM_H[255:128] OH OH Unchanged 
BNDCFGU OH OH OH 
BNDO-BND3 OH OH OH 
1A32_BNDCFGS OH OH OH 
OPMASK OH OH Unchanged 
ZMM_H[511:256] OH OH Unchanged 
ZMMHi16[51 1:0] OH OH Unchanged 
PKRU OH OH Unchanged 
Intel Processor Trace MSRs OH OH” Unchanged 
Time-Stamp Counter OH OH” Unchanged 
IA32_TSC_AUX OH OH Unchanged 
1A32_TSC_ADJUST OH OH Unchanged 
1A32_TSC_DEADLINE OH OH Unchanged 
IA32_SYSENTER_CS/ESP/EIP | OH OH Unchanged 
IA32_EFER O000000000000000H O000000000000000H O000000000000000H 
IA32_STAR/LSTAR OH OH Unchanged 
IA32_FS_BASE/GS_BASE OH OH OH 


Vol. 3A 9-3 


PROCESSOR MANAGEMENT AND INITIALIZATION 


Table 9-1. IA-32 and Intel 64 Processor States Following Power-up, Reset, or INIT (Contd.) 


Register Power up Reset INIT 
IA32_PMCx, OH OH Unchanged 
IA32_PERFEVTSELx 
IA32_FIXED_CTRx, OH OH Unchanged 
IA32_FIXED_CTR_CTRL, 

Global Perf Counter Controls 

Data and Code Cache, TLBs Invalid® Invalid® Unchanged 
Fixed MTRRs Disabled Disabled Unchanged 
Variable MTRRs Disabled Disabled Unchanged 
Machine-Check Banks Undefined Undefined” Unchanged 
Last Branch Record Stack 0 ow Unchanged 
APIC Enabled Enabled Unchanged 
X2APIC Disabled Disabled Unchanged 
IA32_DEBUG_INTERFACE 0 ow Unchanged 


NOTES: 


1. The 10 most-significant bits of the EFLAGS register are undefined following a reset. Software should not depend on the states of 


any of these bits. 


2. The CD and Nw flags are unchanged, bit 4 is set to 1, all other bits are cleared. 

3. Where “n” is the Extended Model Value for the respective processor, and “xx” = don't care. 

4. If Built-In Self-Test (BIST) is invoked on power up or reset, EAX is 0 only if all tests passed. (BIST cannot be invoked during an INIT.) 
5. The state of the x87 FPU and MMxX registers is not changed by the execution of an INIT. 

6. Internal caches are invalid after power-up and RESET, but left unchanged with an INIT. 

W: Warm RESET behavior differs from power-on RESET with details listed in Table 9-2. 


Table 9-2. Variance of RESET Values in Selected Intel Architecture Processors 


State XREF Value Feature Flag or DisplayFamily_DisplayModel Signatures 
Time-Stamp Counter Warm RESET | Unmodified across warm 06_2DH, 06_3€H 
Reset 
Machine-Check Banks Warm RESET | IA32_MCi_Status banks are | 06_2DH, 06_3€H, 06_3FH, 06_4FH, O06_56H 
unmodified across warm 
Reset 
Last Branch Record Stack | Warm RESET | LBR stack MSRs are 06_1AH, 06_1CH, DisplayFamiy= 06 and DisplayModel >1DH 
unmodified across warm 
Reset 
Intel Processor Trace Warm RESET | Clears If CPUID.(EAX=1 4H, ECX=OH):EBX[bit 2] = 1 
MSRs IA32_RTIT_CTL.TraceEn, 
the rest of MSRs are 
unmodified 
IA32_DEBUG_INTERFACE | Warm RESET | Unmodified across warm If CPUID.O1H:ECX[1 1] = 1 


Reset 
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9.1.2 Processor Built-In Self-Test (BIST) 


Hardware may request that the BIST be performed at power-up. The EAX register is cleared (OH) if the processor 
passes the BIST. A nonzero value in the EAX register after the BIST indicates that a processor fault was detected. 
If the BIST is not requested, the contents of the EAX register after a hardware reset is OH. 


The overhead for performing a BIST varies between processor families. For example, the BIST takes approximately 
30 million processor clock periods to execute on the Pentium 4 processor. This clock count is model-specific; Intel 
reserves the right to change the number of periods for any Intel 64 or IA-32 processor, without notification. 


9.1.3 Model and Stepping Information 


Following a hardware reset, the EDX register contains component identification and revision information (see 
Figure 9-2). For example, the model, family, and processor type returned for the first processor in the Intel 
Pentium 4 family is as follows: model (O000B), family (1111B), and processor type (OOB). 


31 28 27 20 19 1615 14 13 12 11 87 43 0 


Extended Extended ; Stepping 
EDX Family Model Family | Model iD 


Processor Type 
Family (1111B for the Pentium 4 Processor Family) 
Model (Beginning with O000B) 


Reserved 


Figure 9-2. Version Information in the EDX Register after Reset 


The stepping ID field contains a unique identifier for the processor’s stepping ID or revision level. The extended 
family and extended model fields were added to the IA-32 architecture in the Pentium 4 processors. 


9.1.4 First Instruction Executed 


The first instruction that is fetched and executed following a hardware reset is located at physical address 
FFFFFFFOH. This address is 16 bytes below the processor’s uppermost physical address. The EPROM containing the 
software-initialization code must be located at this address. 


The address FFFFFFFOH is beyond the 1-MByte addressable range of the processor while in real-address mode. The 
processor is initialized to this starting address as follows. The CS register has two parts: the visible segment 
selector part and the hidden base address part. In real-address mode, the base address is normally formed by 
shifting the 16-bit segment selector value 4 bits to the left to produce a 20-bit base address. However, during a 
hardware reset, the segment selector in the CS register is loaded with FOOOH and the base address is loaded with 
FFFFOOOOH. The starting address is thus formed by adding the base address to the value in the EIP register (that 
is, FFFFOOOO + FFFOH = FFFFFFFOH). 


The first time the CS register is loaded with a new value after a hardware reset, the processor will follow the normal 
rule for address translation in real-address mode (that is, [CS base address = CS segment selector * 16]). To 
ensure that the base address in the CS register remains unchanged until the EPROM based software-initialization 
code is completed, the code must not contain a far jump or far call or allow an interrupt to occur (which would 
cause the CS selector value to be changed). 


9.2 X87 FPU INITIALIZATION 


Software-initialization code can determine the whether the processor contains an x87 FPU by using the CPUID 
instruction. The code must then initialize the x87 FPU and set flags in control register CRO to reflect the state of the 
x87 FPU environment. 
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A hardware reset places the x87 FPU in the state shown in Table 9-1. This state is different from the state the x87 
FPU is placed in following the execution of an FINIT or FNINIT instruction (also shown in Table 9-1). If the x87 FPU 
is to be used, the software-initialization code should execute an FINIT/FNINIT instruction following a hardware 
reset. These instructions, tag all data registers as empty, clear all the exception masks, set the TOP-of-stack value 
to 0, and select the default rounding and precision controls setting (round to nearest and 64-bit precision). 


If the processor is reset by asserting the INIT# pin, the x87 FPU state is not changed. 


9.2.1 Configuring the x87 FPU Environment 


Initialization code must load the appropriate values into the MP, EM, and NE flags of control register CRO. These bits 
are cleared on hardware reset of the processor. Figure 9-3 shows the suggested settings for these flags, depending 
on the IA-32 processor being initialized. Initialization code can test for the type of processor present before setting 
or clearing these flags. 


Table 9-3. Recommended Settings of EM and MP Flags on IA-32 Processors 


EM MP NE IA-32 processor 
1 0 1 Intel486™ SX, Intel386™ DX, and Intel386™ SX processors only, without the presence of a math 
coprocessor. 
0 1 1or0° | Pentium 4, Intel Xeon, P6 family, Pentium, Intel486™ DX, and Intel 487 SX processors, and 
Intel386 DX and Intel386 SX processors when a companion math coprocessor is present. 
0 1 1or0* | More recent Intel 64 or IA-32 processors 
NOTE: 


* The setting of the NE flag depends on the operating system being used. 


The EM flag determines whether floating-point instructions are executed by the x87 FPU (EM is cleared) ora 
device-not-available exception (#NM) is generated for all floating-point instructions so that an exception handler 
can emulate the floating-point operation (EM = 1). Ordinarily, the EM flag is cleared when an x87 FPU or math 
coprocessor is present and set if they are not present. If the EM flag is set and no x87 FPU, math coprocessor, or 
floating-point emulator is present, the processor will hang when a floating-point instruction is executed. 


The MP flag determines whether WAIT/FWAIT instructions react to the setting of the TS flag. If the MP flag is clear, 
WAIT/FWAIT instructions ignore the setting of the TS flag; if the MP flag is set, they will generate a device-not- 
available exception (#NM) if the TS flag is set. Generally, the MP flag should be set for processors with an inte- 
grated x87 FPU and clear for processors without an integrated x87 FPU and without a math coprocessor present. 
However, an operating system can choose to save the floating-point context at every context switch, in which case 
there would be no need to set the MP bit. 


Table 2-2 shows the actions taken for floating-point and WAIT/FWAIT instructions based on the settings of the EM, 
MP, and TS flags. 


The NE flag determines whether unmasked floating-point exceptions are handled by generating a floating-point 
error exception internally (NE is set, native mode) or through an external interrupt (NE is cleared). In systems 
where an external interrupt controller is used to invoke numeric exception handlers (such as MS-DOS-based 
systems), the NE bit should be cleared. 


9.2.2 Setting the Processor for x87 FPU Software Emulation 


Setting the EM flag causes the processor to generate a device-not-available exception (#NM) and trap to a software 
exception handler whenever it encounters a floating-point instruction. (Table 9-3 shows when it is appropriate to 
use this flag.) Setting this flag has two functions: 


¢ It allows x87 FPU code to run on an IA-32 processor that has neither an integrated x87 FPU nor is connected to 
an external math coprocessor, by using a floating-point emulator. 


¢ It allows floating-point code to be executed using a special or nonstandard floating-point emulator, selected for 
a particular application, regardless of whether an x87 FPU or math coprocessor is present. 
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To emulate floating-point instructions, the EM, MP, and NE flag in control register CRO should be set as shown in 
Table 9-4, 


Table 9-4. Software Emulation Settings of EM, MP, and NE Flags 


CRO Bit Value 
EM 1 
MP 0 
NE 1 


Regardless of the value of the EM bit, the Intel486 SX processor generates a device-not-available exception (#NM) 
upon encountering any floating-point instruction. 


9.3 CACHE ENABLING 


IA-32 processors (beginning with the Intel486 processor) and Intel 64 processors contain internal instruction and 
data caches. These caches are enabled by clearing the CD and NW flags in control register CRO. (They are set 
during a hardware reset.) Because all internal cache lines are invalid following reset initialization, it is not neces- 
sary to invalidate the cache before enabling caching. Any external caches may require initialization and invalidation 
using a system-specific initialization and invalidation code sequence. 


Depending on the hardware and operating system or executive requirements, additional configuration of the 
processor's caching facilities will probably be required. Beginning with the Intel486 processor, page-level caching 
can be controlled with the PCD and PWT flags in page-directory and page-table entries. Beginning with the P6 
family processors, the memory type range registers (MTRRs) control the caching characteristics of the regions of 
physical memory. (For the Intel486 and Pentium processors, external hardware can be used to control the caching 
characteristics of regions of physical memory.) See Chapter 11, “Memory Cache Control,” for detailed information 
on configuration of the caching facilities in the Pentium 4, Intel Xeon, and P6 family processors and system 
memory. 


9.4 MODEL-SPECIFIC REGISTERS (MSRS) 


Most IA-32 processors (starting from Pentium processors) and Intel 64 processors contain a model-specific regis- 
ters (MSRs). A given MSR may not be supported across all families and models for Intel 64 and IA-32 processors. 
Some MSRs are designated as architectural to simplify software programming; a feature introduced by an architec- 
tural MSR is expected to be supported in future processors. Non-architectural MSRs are not guaranteed to be 
supported or to have the same functions on future processors. 


MSRs that provide control for a number of hardware and software-related features, include: 
¢ Performance-monitoring counters (see Chapter 18, “Performance Monitoring”). 


° Debug extensions (see Chapter 17, “Debug, Branch Profile, TSC, and Intel® Resource Director Technology 
(Intel® RDT) Features”). 


® = Machine-check exception capability and its accompanying machine-check architecture (see Chapter 15, 
“Machine-Check Architecture”). 


¢ MTRRs (see Section 11.11, “Memory Type Range Registers (MTRRs)”). 

¢ Thermal and power management. 

¢ Instruction-specific support (for example: SYSENTER, SYSEXIT, SWAPGS, etc.). 

¢ Processor feature/mode support (for example: IA32_EFER, IA32_FEATURE_CONTROL). 
The MSRs can be read and written to using the RDMSR and WRMSR instructions, respectively. 


When performing software initialization of an IA-32 or Intel 64 processor, many of the MSRs will need to be initial- 
ized to set up things like performance-monitoring events, run-time machine checks, and memory types for phys- 
ical memory. 
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Lists of available performance-monitoring events can be found at: https://perfmon-events.intel.com/, and lists of 
available MSRs are given in Chapter 2, “Model-Specific Registers (MSRs)” in the Intel® 64 and IA-32 Architectures 
Software Developer’s Manual, Volume 4. The references earlier in this section show where the functions of the 
various groups of MSRs are described in this manual. 


9.5 MEMORY TYPE RANGE REGISTERS (MTRRS) 


Memory type range registers (MTRRs) were introduced into the IA-32 architecture with the Pentium Pro processor. 
They allow the type of caching (or no caching) to be specified in system memory for selected physical address 
ranges. They allow memory accesses to be optimized for various types of memory such as RAM, ROM, frame buffer 
memory, and memory-mapped I/O devices. 


In general, initializing the MTRRs is normally handled by the software initialization code or BIOS and is not an oper- 
ating system or executive function. At the very least, all the MTRRs must be cleared to 0, which selects the 
uncached (UC) memory type. See Section 11.11, “Memory Type Range Registers (MTRRs),” for detailed informa- 
tion on the MTRRs. 


9.6 INITIALIZING SSE/SSE2/SSE3/SSSE3 EXTENSIONS 


For processors that contain SSE/SSE2/SSE3/SSSE3 extensions, steps must be taken when initializing the 
processor to allow execution of these instructions. 


1. Check the CPUID feature flags for the presence of the SSE/SSE2/SSE3/SSSE3 extensions (respectively: EDX 
bits 25 and 26, ECX bit 0 and 9) and support for the FXSAVE and FXRSTOR instructions (EDX bit 24). Also check 
for support for the CLFLUSH instruction (EDX bit 19). The CPUID feature flags are loaded in the EDX and ECX 
registers when the CPUID instruction is executed with a 1 in the EAX register. 


2. Set the OSFXSR flag (bit 9 in control register CR4) to indicate that the operating system supports saving and 
restoring the SSE/SSE2/SSE3/SSSE3 execution environment (XMM and MXCSR registers) with the FXSAVE and 
FXRSTOR instructions, respectively. See Section 2.5, “Control Registers,” for a description of the OSFXSR flag. 


3. Set the OSXMMEXCPT flag (bit 10 in control register CR4) to indicate that the operating system supports the 
handling of SSE/SSE2/SSE3 SIMD floating-point exceptions (#XM). See Section 2.5, “Control Registers,” for a 
description of the OSXMMEXCPT flag. 


4. Set the mask bits and flags in the MXCSR register according to the mode of operation desired for 
SSE/SSE2/SSE3 SIMD floating-point instructions. See “MXCSR Control and Status Register” in Chapter 10, 
“Programming with Streaming SIMD Extensions (SSE),” of the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1, for a detailed description of the bits and flags in the MXCSR register. 


9.7 SOFTWARE INITIALIZATION FOR REAL-ADDRESS MODE OPERATION 


Following a hardware reset (either through a power-up or the assertion of the RESET# pin) the processor is placed 
in real-address mode and begins executing software initialization code from physical address FFFFFFFOH. Software 
initialization code must first set up the necessary data structures for handling basic system functions, such as a 
real-mode IDT for handling interrupts and exceptions. If the processor is to remain in real-address mode, software 
must then load additional operating-system or executive code modules and data structures to allow reliable execu- 
tion of application programs in real-address mode. 


If the processor is going to operate in protected mode, software must load the necessary data structures to operate 
in protected mode and then switch to protected mode. The protected-mode data structures that must be loaded 
are described in Section 9.8, “Software Initialization for Protected-Mode Operation.” 


9.7.1 Real-Address Mode IDT 


In real-address mode, the only system data structure that must be loaded into memory is the IDT (also called the 
“interrupt vector table”). By default, the address of the base of the IDT is physical address OH. This address can be 
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changed by using the LIDT instruction to change the base address value in the IDTR. Software initialization code 
needs to load interrupt- and exception-handler pointers into the IDT before interrupts can be enabled. 


The actual interrupt- and exception-handler code can be contained either in EPROM or RAM; however, the code 
must be located within the 1-MByte addressable range of the processor in real-address mode. If the handler code 
is to be stored in RAM, it must be loaded along with the IDT. 


9.7.2 NMI Interrupt Handling 


The NMI interrupt is always enabled (except when multiple NMIs are nested). If the IDT and the NMI interrupt 
handler need to be loaded into RAM, there will be a period of time following hardware reset when an NMI interrupt 
cannot be handled. During this time, hardware must provide a mechanism to prevent an NMI interrupt from halting 
code execution until the IDT and the necessary NMI handler software is loaded. Here are two examples of how 
NMIs can be handled during the initial states of processor initialization: 


¢ Asimple IDT and NMI interrupt handler can be provided in EPROM. This allows an NMI interrupt to be handled 
immediately after reset initialization. 


¢ The system hardware can provide a mechanism to enable and disable NMIs by passing the NMI# signal through 
an AND gate controlled by a flag in an I/O port. Hardware can clear the flag when the processor is reset, and 
software can set the flag when it is ready to handle NMI interrupts. 


9.8 SOFTWARE INITIALIZATION FOR PROTECTED-MODE OPERATION 


The processor is placed in real-address mode following a hardware reset. At this point in the initialization process, 
some basic data structures and code modules must be loaded into physical memory to support further initialization 
of the processor, as described in Section 9.7, “Software Initialization for Real-Address Mode Operation.” Before the 
processor can be switched to protected mode, the software initialization code must load a minimum number of 
protected mode data structures and code modules into memory to support reliable operation of the processor in 
protected mode. These data structures include the following: 


° AIDT. 
° AGDT. 
° ATSS. 


¢ (Optional) An LDT. 

° If paging is to be used, at least one page directory and one page table. 

¢ Acode segment that contains the code to be executed when the processor switches to protected mode. 
¢ One or more code modules that contain the necessary interrupt and exception handlers. 


Software initialization code must also initialize the following system registers before the processor can be switched 
to protected mode: 


¢ The GDTR. 


¢ (Optional.) The IDTR. This register can also be initialized immediately after switching to protected mode, prior 
to enabling interrupts. 


* Control registers CR1 through CR4. 
¢ (Pentium 4, Intel Xeon, and P6 family processors only.) The memory type range registers (MTRRs). 


With these data structures, code modules, and system registers initialized, the processor can be switched to 
protected mode by loading control register CRO with a value that sets the PE flag (bit 0). 


9.8.1 Protected-Mode System Data Structures 


The contents of the protected-mode system data structures loaded into memory during software initialization, 
depend largely on the type of memory management the protected-mode operating-system or executive is going to 
support: flat, flat with paging, segmented, or segmented with paging. 
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To implement a flat memory model without paging, software initialization code must at a minimum load a GDT with 
one code and one data-segment descriptor. A null descriptor in the first GDT entry is also required. The stack can 
be placed in a normal read/write data segment, so no dedicated descriptor for the stack is required. A flat memory 
model with paging also requires a page directory and at least one page table (unless all pages are 4 MBytes in 
which case only a page directory is required). See Section 9.8.3, “Initializing Paging.” 


Before the GDT can be used, the base address and limit for the GDT must be loaded into the GDTR register using 
an LGDT instruction. 


A multi-segmented model may require additional segments for the operating system, as well as segments and LDTs 
for each application program. LDTs require segment descriptors in the GDT. Some operating systems allocate new 
segments and LDTs as they are needed. This provides maximum flexibility for handling a dynamic programming 
environment. However, many operating systems use a single LDT for all tasks, allocating GDT entries in advance. 
An embedded system, such as a process controller, might pre-allocate a fixed number of segments and LDTs for a 
fixed number of application programs. This would be a simple and efficient way to structure the software environ- 
ment of a real-time system. 


9.8.2 Initializing Protected-Mode Exceptions and Interrupts 


Software initialization code must at a minimum load a protected-mode IDT with gate descriptor for each exception 
vector that the processor can generate. If interrupt or trap gates are used, the gate descriptors can all point to the 
same code segment, which contains the necessary exception handlers. If task gates are used, one TSS and accom- 
panying code, data, and task segments are required for each exception handler called with a task gate. 


If hardware allows interrupts to be generated, gate descriptors must be provided in the IDT for one or more inter- 
rupt handlers. 


Before the IDT can be used, the base address and limit for the IDT must be loaded into the IDTR register using an 
LIDT instruction. This operation is typically carried out immediately after switching to protected mode. 


9.8.3 Initializing Paging 


Paging is controlled by the PG flag in control register CRO. When this flag is clear (its state following a hardware 
reset), the paging mechanism is turned off; when it is set, paging is enabled. Before setting the PG flag, the 
following data structures and registers must be initialized: 


¢ Software must load at least one page directory and one page table into physical memory. The page table can 
be eliminated if the page directory contains a directory entry pointing to itself (here, the page directory and 
page table reside in the same page), or if only 4-MByte pages are used. 


* Control register CR3 (also called the PDBR register) is loaded with the physical base address of the page 
directory. 


¢ (Optional) Software may provide one set of code and data descriptors in the GDT or in an LDT for supervisor 
mode and another set for user mode. 


With this paging initialization complete, paging is enabled and the processor is switched to protected mode at the 
same time by loading control register CRO with an image in which the PG and PE flags are set. (Paging cannot be 
enabled before the processor is switched to protected mode.) 


9.8.4 Initializing Multitasking 


If the multitasking mechanism is not going to be used and changes between privilege levels are not allowed, it is 
not necessary load a TSS into memory or to initialize the task register. 


If the multitasking mechanism is going to be used and/or changes between privilege levels are allowed, software 
initialization code must load at least one TSS and an accompanying TSS descriptor. (A TSS is required to change 
privilege levels because pointers to the privileged-level 0, 1, and 2 stack segments and the stack pointers for these 
stacks are obtained from the TSS.) TSS descriptors must not be marked as busy when they are created; they 
should be marked busy by the processor only as a side-effect of performing a task switch. As with descriptors for 
LDTs, TSS descriptors reside in the GDT. 
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After the processor has switched to protected mode, the LTR instruction can be used to load a segment selector for 
a TSS descriptor into the task register. This instruction marks the TSS descriptor as busy, but does not perform a 
task switch. The processor can, however, use the TSS to locate pointers to privilege-level 0, 1, and 2 stacks. The 
segment selector for the TSS must be loaded before software performs its first task switch in protected mode, 
because a task switch copies the current task state into the TSS. 


After the LTR instruction has been executed, further operations on the task register are performed by task 
switching. As with other segments and LDTs, TSSs and TSS descriptors can be either pre-allocated or allocated as 
needed. 


9.8.5 Initializing |A-32e Mode 


On Intel 64 processors, the IA32_EFER MSR is cleared on system reset. The operating system must be in protected 
mode with paging enabled before attempting to initialize IA-32e mode. IA-32e mode operation also requires phys- 
ical-address extensions with four or five levels of enhanced paging structures (see Section 4.5, “4-Level Paging and 
5-Level Paging”). 


Operating systems should follow this sequence to initialize IA-32e mode: 


1. Starting from protected mode, disable paging by setting CRO.PG = 0. Use the MOV CRO instruction to disable 
paging (the instruction must be located in an identity-mapped page). 


2. Enable physical-address extensions (PAE) by setting CR4.PAE = 1. Failure to enable PAE will result in a #GP 
fault when an attempt is made to initialize IA-32e mode. 


3. Load CR3 with the physical base address of the Level 4 page map table (PML4) or Level 5 page map table 
(PMLS). 


Enable IA-32e mode by setting IA32_EFER.LME = 1. 


5. Enable paging by setting CRO.PG = 1. This causes the processor to set the IA32_EFER.LMA bit to 1. The MOV 
CRO instruction that enables paging and the following instructions must be located in an identity-mapped page 
(until such time that a branch to non-identity mapped pages can be effected). 


64-bit mode paging structures must be located in the first 4 GBytes of physical-address space prior to activating 
IA-32e mode. This is necessary because the MOV CR3 instruction used to initialize the page-directory base must be 
executed in legacy mode prior to activating IA-32e mode (setting CRO.PG = 1 to enable paging). Because MOV CR3 
is executed in protected mode, only the lower 32 bits of the register are written, limiting the table location to the 
low 4 GBytes of memory. Software can relocate the page tables anywhere in physical memory after IA-32e mode 
is activated. 


The processor performs 64-bit mode consistency checks whenever software attempts to modify any of the enable 
bits directly involved in activating IA-32e mode (IA32_EFER.LME, CRO.PG, and CR4.PAE). It will generate a general 
protection fault (#GP) if consistency checks fail. 64-bit mode consistency checks ensure that the processor does 
not enter an undefined mode or state with unpredictable behavior. 


64-bit mode consistency checks fail in the following circumstances: 
¢ An attempt is made to enable or disable IA-32e mode while paging is enabled. 


e JIA-32e mode is enabled and an attempt is made to enable paging prior to enabling physical-address extensions 
(PAE). 


¢ IA-32e mode is active and an attempt is made to disable physical-address extensions (PAE). 
e If the current CS has the L-bit set on an attempt to activate IA-32e mode. 
e If the TR contains a 16-bit TSS on an attempt to activate IA-32e mode. 


9.8.5.1 IA-32e Mode System Data Structures 


After activating IA-32e mode, the system-descriptor-table registers (GDTR, LDTR, IDTR, TR) continue to reference 
legacy protected-mode descriptor tables. Tables referenced by the descriptors all reside in the lower 4 GBytes of 
linear-address space. After activating IA-32e mode, 64-bit operating-systems should use the LGDT, LLDT, LIDT, 
and LTR instructions to load the system-descriptor-table registers with references to 64-bit descriptor tables. 
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9.8.5.2 IA-32e Mode Interrupts and Exceptions 


Software must not allow exceptions or interrupts to occur between the time IA-32e mode is activated and the 
update of the interrupt-descriptor-table register (IDTR) that establishes references to a 64-bit interrupt-descriptor 
table (IDT). This is because the IDT remains in legacy form immediately after IA-32e mode is activated. 


If an interrupt or exception occurs prior to updating the IDTR, a legacy 32-bit interrupt gate will be referenced and 
interpreted as a 64-bit interrupt gate with unpredictable results. External interrupts can be disabled by using the 
CLI instruction. 


Non-maskable interrupts (NMI) must be disabled using external hardware. 


9.8.5.3 64-bit Mode and Compatibility Mode Operation 


IA-32e mode uses two code segment-descriptor bits (CS.L and CS.D, see Figure 3-8) to control the operating modes 
after IA-32e mode is initialized. If CS.L = 1 and CS.D = 0, the processor is running in 64-bit mode. With this 
encoding, the default operand size is 32 bits and default address size is 64 bits. Using instruction prefixes, operand 
size can be changed to 64 bits or 16 bits; address size can be changed to 32 bits. 


When IA-32e mode is active and CS.L = 0, the processor operates in compatibility mode. In this mode, CS.D 
controls default operand and address sizes exactly as it does in the IA-32 architecture. Setting CS.D = 1 specifies 
default operand and address size as 32 bits. Clearing CS.D to 0 specifies default operand and address size as 16 
bits (the CS.L = 1, CS.D = 1 bit combination is reserved). 


Compatibility mode execution is selected on a code-segment basis. This mode allows legacy applications to coexist 
with 64-bit applications running in 64-bit mode. An operating system running in IA-32e mode can execute existing 
16-bit and 32-bit applications by clearing their code-segment descriptor’s CS.L bit to 0. 


In compatibility mode, the following system-level mechanisms continue to operate using the IA-32e-mode archi- 
tectural semantics: 


¢ Linear-to-physical address translation uses the 64-bit mode extended page-translation mechanism. 
¢* Interrupts and exceptions are handled using the 64-bit mode mechanisms. 


¢ System calls (calls through call gates and SYSENTER/SYSEXIT) are handled using the IA-32e mode 
mechanisms. 


9.8.5.4 Switching Out of IA-32e Mode Operation 


To return from IA-32e mode to paged-protected mode operation operating systems must use the following 
sequence: 


1. Switch to compatibility mode. 


2. Deactivate IA-32e mode by clearing CRO.PG = 0. This causes the processor to set IA32_EFER.LMA = 0. The 
MOV CRO instruction used to disable paging and subsequent instructions must be located in an identity-mapped 
page. 

Load CR3 with the physical base address of the legacy page-table-directory base address. 

Disable IA-32e mode by setting IA32_EFER.LME = 0. 

Enable legacy paged-protected mode by setting CRO.PG = 1 


A branch instruction must follow the MOV CRO that enables paging. Both the MOV CRO and the branch 
instruction must be located in an identity-mapped page. 


OO: Bw 


Registers only available in 64-bit mode (R8-R15 and XMM8-XMM15) are preserved across transitions from 64-bit 
mode into compatibility mode then back into 64-bit mode. However, values of R8-R15 and XMM8-XMM15 are unde- 
fined after transitions from 64-bit mode through compatibility mode to legacy or real mode and then back through 
compatibility mode to 64-bit mode. 
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9.9 MODE SWITCHING 


To use the processor in protected mode after hardware or software reset, a mode switch must be performed from 
real-address mode. Once in protected mode, software generally does not need to return to real-address mode. To 
run software written to run in real-address mode (8086 mode), it is generally more convenient to run the software 
in virtual-8086 mode, than to switch back to real-address mode. 


9.9.1 Switching to Protected Mode 


Before switching to protected mode from real mode, a minimum set of system data structures and code modules 
must be loaded into memory, as described in Section 9.8, “Software Initialization for Protected-Mode Operation.” 
Once these tables are created, software initialization code can switch into protected mode. 


Protected mode is entered by executing a MOV CRO instruction that sets the PE flag in the CRO register. (In the 
same instruction, the PG flag in register CRO can be set to enable paging.) Execution in protected mode begins with 
a CPL of 0. 


Intel 64 and IA-32 processors have slightly different requirements for switching to protected mode. To ensure 
upwards and downwards code compatibility with Intel 64 and IA-32 processors, we recommend that you follow 
these steps: 


1. Disable interrupts. A CLI instruction disables maskable hardware interrupts. NMI interrupts can be disabled 
with external circuitry. (Software must guarantee that no exceptions or interrupts are generated during the 
mode switching operation.) 


Execute the LGDT instruction to load the GDTR register with the base address of the GDT. 
Execute a MOV CRO instruction that sets the PE flag (and optionally the PG flag) in control register CRO. 


Immediately following the MOV CRO instruction, execute a far JMP or far CALL instruction. (This operation is 
typically a far jump or call to the next instruction in the instruction stream.) 


5. The JMP or CALL instruction immediately after the MOV CRO instruction changes the flow of execution and 
serializes the processor. 


6. If paging is enabled, the code for the MOV CRO instruction and the JMP or CALL instruction must come from a 
page that is identity mapped (that is, the linear address before the jump is the same as the physical address 
after paging and protected mode is enabled). The target instruction for the JMP or CALL instruction does not 
need to be identity mapped. 


7. Ifalocal descriptor table is going to be used, execute the LLDT instruction to load the segment selector for the 
LDT in the LDTR register. 


8. Execute the LTR instruction to load the task register with a segment selector to the initial protected-mode task 
or to a writable area of memory that can be used to store TSS information on a task switch. 


9. After entering protected mode, the segment registers continue to hold the contents they had in real-address 
mode. The JMP or CALL instruction in step 4 resets the CS register. Perform one of the following operations to 
update the contents of the remaining segment registers. 


— Reload segment registers DS, SS, ES, FS, and GS. If the ES, FS, and/or GS registers are not going to be 
used, load them with a null selector. 


— Perform a JMP or CALL instruction to a new task, which automatically resets the values of the segment 
registers and branches to a new code segment. 


10. Execute the LIDT instruction to load the IDTR register with the address and limit of the protected-mode IDT. 


11. Execute the STI instruction to enable maskable hardware interrupts and perform the necessary hardware 
operation to enable NMI interrupts. 


Random failures can occur if other instructions exist between steps 3 and 4 above. Failures will be readily seen in 
some situations, such as when instructions that reference memory are inserted between steps 3 and 4 while in 
system management mode. 
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9.9.2 Switching Back to Real-Address Mode 


The processor switches from protected mode back to real-address mode if software clears the PE bit in the CRO 
register with a MOV CRO instruction. A procedure that re-enters real-address mode should perform the following 
steps: 


1. Disable interrupts. A CLI instruction disables maskable hardware interrupts. NMI interrupts can be disabled 
with external circuitry. 


2. If paging is enabled, perform the following operations: 


— Transfer program control to linear addresses that are identity mapped to physical addresses (that is, linear 
addresses equal physical addresses). 


— Ensure that the GDT and IDT are in identity mapped pages. 
— Clear the PG bit in the CRO register. 
— Move OH into the CR3 register to flush the TLB. 


3. Transfer program control to a readable segment that has a limit of 64 KBytes (FFFFH). This operation loads the 
CS register with the segment limit required in real-address mode. 


4. Load segment registers SS, DS, ES, FS, and GS with a selector for a descriptor containing the following values, 
which are appropriate for real-address mode: 


— Limit = 64 KBytes (OFFFFH) 
— Byte granular (G = 0) 

— Expand up (E = 0) 

— Writable (W = 1) 

— Present (P = 1) 

— Base = any value 


The segment registers must be loaded with non-null segment selectors or the segment registers will be 
unusable in real-address mode. Note that if the segment registers are not reloaded, execution continues using 
the descriptor attributes loaded during protected mode. 


5. Execute an LIDT instruction to point to a real-address mode interrupt table that is within the 1-MByte real- 
address mode address range. 


Clear the PE flag in the CRO register to switch to real-address mode. 


Execute a far JMP instruction to jump to a real-address mode program. This operation flushes the instruction 
queue and loads the appropriate base-address value in the CS register. 


8. Load the SS, DS, ES, FS, and GS registers as needed by the real-address mode code. If any of the registers are 
not going to be used in real-address mode, write Os to them. 


9. Execute the STI instruction to enable maskable hardware interrupts and perform the necessary hardware 
operation to enable NMI interrupts. 


NOTE 


All the code that is executed in steps 1 through 9 must be in a single page and the linear addresses 
in that page must be identity mapped to physical addresses. 


9.10 INITIALIZATION AND MODE SWITCHING EXAMPLE 


This section provides an initialization and mode switching example that can be incorporated into an application. 
This code was originally written to initialize the Intel386 processor, but it will execute successfully on the Pentium 
4, Intel Xeon, P6 family, Pentium, and Intel486 processors. The code in this example is intended to reside in EPROM 
and to run following a hardware reset of the processor. The function of the code is to do the following: 


¢ Establish a basic real-address mode operating environment. 
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Load the necessary protected-mode system data structures into RAM. 


Load the system registers with the necessary pointers to the data structures and the appropriate flag settings 
for protected-mode operation. 


Switch the processor to protected mode. 


Figure 9-3 shows the physical memory layout for the processor following a hardware reset and the starting point of 
this example. The EPROM that contains the initialization code resides at the upper end of the processor’s physical 
memory address range, starting at address FFFFFFFFH and going down from there. The address of the first instruc- 
tion to be executed is at FFFFFFFOH, the default starting address for the processor following a hardware reset. 


The main steps carried out in this example are summarized in Table 9-5. The source listing for the example (with 
the filename STARTUP.ASM) is given in Example 9-1. The line numbers given in Table 9-5 refer to the source listing. 


The following are some additional notes concerning this example: 


When the processor is switched into protected mode, the original code segment base-address value of 
FFFFOOOOH (located in the hidden part of the CS register) is retained and execution continues from the current 
offset in the EIP register. The processor will thus continue to execute code in the EPROM until a far jump or call 
is made to a new code segment, at which time, the base address in the CS register will be changed. 


Maskable hardware interrupts are disabled after a hardware reset and should remain disabled until the 


necessary interrupt handlers have been installed. The NMI interrupt is not disabled following a reset. The NMI# 
pin must thus be inhibited from being asserted until an NMI handler has been loaded and made available to the 


processor. 
The use of a temporary GDT allows simple transfer of tables from the EPROM to anywhere in the RAM area. A 
GDT entry is constructed with its base pointing to address 0 and a limit of 4 GBytes. When the DS and ES 
registers are loaded with this descriptor, the temporary GDT is no longer needed and can be replaced by the 
application GDT. 

This code loads one TSS and no LDTs. If more TSSs exist in the application, they must be loaded into RAM. If 
there are LDTs they may be loaded as well. 


After Reset 
FFFF FFFFH A 
[CS.BASE+EIP] > — 4 FFFF wal 
64K EPROM 
EIP = 0000 FFFOH 
CS.BASE = FFFF 0000H FFFF 0000H 
DS.BASE = 0H 
ES.BASE = 0H 
SS.BASE = 0H 
ESP = 0H 
Vv 
S S 
[SP, DS, SS, ES] co 0) 


Figure 9-3. Processor State After Reset 
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Table 9-5. Main Initialization Steps in STARTUP.ASM Source Listing 


STARTUP.ASM Line Description 
Numbers 
From To 
157 157 Jump (short) to the entry code in the EPROM 
162 169 Construct a temporary GDT in RAM with one entry: 
- oa data segment, base = 0, limit = 4 GBytes 
171 172 Load the GDTR to point to the temporary GDT 
174 177 Load CRO with PE flag set to switch to protected mode 
179 181 Jump near to clear real mode instruction queue 
184 186 Load DS, ES registers with GDT[1] descriptor, so both point to the entire physical memory space 
188 195 Perform specific board initialization that is imposed by the new protected mode 
196 218 Copy the application's GDT from ROM into RAM 
220 238 Copy the application's IDT from ROM into RAM 
241 243 Load application’s GDTR 
244 245 Load application's IDTR 
247 261 Copy the application's TSS from ROM into RAM 
263 267 Update TSS descriptor and other aliases in GDT (GDT alias or IDT alias) 
277 277 Load the task register (without task switch) using LTR instruction 
282 286 Load SS, ESP with the value found in the application's TSS 
287 287 Push EFLAGS value found in the application’s TSS 
288 288 Push CS value found in the application’s TSS 
289 289 Push EIP value found in the application’s TSS 
290 293 Load DS, ES with the value found in the application’s TSS 
296 296 Perform IRET; pop the above values and enter the application code 


9.10.1 Assembler Usage 


In this example, the Intel assembler ASM386 and build tools BLD386 are used to assemble and build the initializa- 
tion code module. The following assumptions are used when using the Intel ASM386 and BLD386 tools. 


° The ASM386 will generate the right operand size opcodes according to the code-segment attribute. The 
attribute is assigned either by the ASM386 invocation controls or in the code-segment definition. 


¢ If a code segment that is going to run in real-address mode is defined, it must be set to a USE 16 attribute. If 
a 32-bit operand is used in an instruction in this code segment (for example, MOV EAX, EBX), the assembler 
automatically generates an operand prefix for the instruction that forces the processor to execute a 32-bit 
operation, even though its default code-segment attribute is 16-bit. 


¢ Intel's ASM386 assembler allows specific use of the 16- or 32-bit instructions, for example, LGDTW, LGDTD, 
IRETD. If the generic instruction LGDT is used, the default- segment attribute will be used to generate the right 
opcode. 


9.10.2 STARTUP.ASM Listing 


Example 9-1 provides high-level sample code designed to move the processor into protected mode. This listing 
does not include any opcode and offset information. 
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Example 9-1. STARTUP.ASM 


MS-DOS* 5.0(045-N) 386(TM) MACRO ASSEMBLER STARTUP 09:44:51 08/19/92 PAGE 1 


MS-DOS 5.0(045-N) 386(TM) MACRO ASSEMBLER V4.0, ASSEMBLY OF MODULE STARTUP 
OBJECT MODULE PLACED IN startup.obj 
ASSEMBLER INVOKED BY: f£:\386tools\ASM386.EXE startup.a58 pw (132 ) 


LINE 


OarAI A HF WN HP 


SOURCE 
NAME STARTUP 
i ASSUMPTIONS: 


; 1. The bottom 64K of memory is ram, and can be used for 
; scratch space by this module. 


i 2. The system has sufficient free usable ram to copy the 
; initial GDT, IDT, and TSS 


SP BP RE Pa Ro Ee EO RE EOP Pe Pa EP RS PO eR a a RR Be 
; configuration data - must match with build definition 
CS_ BASE EQU OFFFFOOOOH 


; CS BASE is the linear address of the segment STARTUP CODE 
; - this is specified in the build language file 


RAM START EQU 400H 


; RAM _START is the start of free, usable ram in the linear 
; memory space. The GDT, IDT, and initial TSS will be 
; copied above this space, and a small data segment will be 
; Giscarded at this linear address. The 32-bit word at 
; RAM_START will contain the linear address of the first 
; free byte above the copied tables - this may be useful if 
; a memory manager is used. 


TSS INDEX EQU 10 


; TSS_INDEX is the index of the TSS of the first task to 
; run after startup 


prrrrrrrrr ere 


pococc conn een nee ----------- STRUCTURES and EQU --------------- 
; structures for system data 


; TSS structure 
TASK STATE STRUC 
link DW ? 


Vol. 3A 9-17 


PROCESSOR MANAGEMENT AND INITIALIZATION 


46 link h DW? 
47 ESPO DD ? 
48 SSO DW ? 
49 sso_h DW ? 
50 ESP1 DD ? 
51 SS1 DW ? 
52 ssilh DW ? 
53 ESP2 DD ? 
54 SS2 DW ? 
55 SS2_h DW ? 
56 CR3_reg DD ? 
57 EIP_ reg DD ? 
58 EFLAGS regDD ? 
59 EAX_reg DD ? 
60 ECX_reg DD ? 
61 EDX reg DD ? 
62 EBX reg DD ? 
63 ESP_reg DD ? 
64 EBP_ reg DD ? 
65 ESI_reg DD ? 
66 EDI_reg DD ? 
67 ES_reg DW ? 
68 ES h DW ? 
69 CS_ reg DW ? 
70 cS h DW ? 
71 SS_ reg DW ? 
72 ssh DW ? 
73 DS_reg DW ? 
74 DS _h DW ? 
75 FS reg DW ? 
76 FS h DW ? 
77 GS_reg DW ? 
78 GS_h DW ? 
ag LDT_ reg DW ? 
80 LDT h DW ? 
81 TRAP _reg DW ? 
82 IO_map_baseDW ? 
83 TASK STATE ENDS 
84 

85 ; basic structure of a descriptor 
86 DESC STRUC 

87 lim_0.15 DW ? 
88 bas 0.15 DW ? 
89 bas_16 23DB ? 
90 access DB ? 
91 gran DB ? 
92 bas 24 31DB ? 
93 DESC ENDS 

94 

95 j; structure for use with LGDT and LIDT instructions 
96 TABLE REG STRUC 
97 table limDW ? 
98 table linearDD ? 
99 TABLE REG ENDS 
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; offset of GDT and IDT descriptors in builder generated GDT 
GDT_DESC_ OFF EQU 1*SIZE (DESC) 
IDT_DESC_ OFF EQU 2*SIZE (DESC) 


; equates for building temporary GDT in RAM 


LINEAR SEL EQU 1*SIZE (DESC) 
LINEAR PROTO LO EQU OOOOOFFFFH ; LINEAR ALTAS 
LINEAR PROTO HI EQU OO0OCF9200H 


; Protection Enable Bit in CRO 
PE BIT EQU 1B 


; Initially, this data segment starts at linear 0, according 
; to the processor’s power-up state. 


STARTUP DATA SEGMENT RW 


free mem_linear_ base LABEL DWORD 

TEMP _GDT LABEL BYTE ; must be first in segment 
TEMP _GDT NULL DESC DESC <> 

TEMP _GDT LINEAR DESC DESC <> 


; scratch areas for LGDT and LIDT instructions 
TEMP _GDT SCRATCH TABLE REG <> 


APP GDT_ RAM TABLE REG <> 
APP IDT RAM TABLE REG <> 
; align end_data 

fill DW ? 


; last thing in this segment - should be on a dword boundary 
end_ data LABEL BYTE 


STARTUP DATA ENDS 


P Sec Seoe sso teeta tse oe cee CODE SEGMENT---------------------- 
STARTUP_CODE SEGMENT ER PUBLIC USE16 


; filled in by builder 
PUBLIC GDT_EPROM 
GDT_EPROM TABLE REG <> 


; filled in by builder 
PUBLIC IDT_EPROM 
IDT_EPROM TABLE REG <> 


; entry point into startup code - the bootstrap will vector 
; here with a near JMP generated by the builder. This 
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154 ; label must be in the top 64K of linear memory. 
155: 

156 PUBLIC STARTUP 

157 STARTUP: 

158 

159 ; DS,ES address the bottom 64K of flat linear memory 
160 ASSUME DS:STARTUP_DATA, ES:STARTUP_DATA 

161 ; See Figure 9-4 

162 ; load GDTR with temporary GDT 

163 LEA EBX,TEMP_GDT ; build the TEMP GDT in low ram, 
164 MOV DWORD PTR [EBX] ,0 ; where we can address 
165 MOV DWORD PTR [EBX] +4,0 

166 MOV DWORD PTR [EBX]+8, LINEAR PROTO LO 
167 MOV DWORD PTR [EBX]+12, LINEAR PROTO HI 
168 MOV TEMP _GDT_scratch.table linear, EBX 
169 MOV TEMP GDT_scratch.table lim,15 

170 

171 DB 66H; execute a 32 bit LGDT 

172 LGDT TEMP _GDT_ scratch 

173 

174 ; enter protected mode 

175 MOV EBX, CRO 

176 OR EBX, PE BIT 

177 MOV CRO, EBX 

178 

LTD ; clear prefetch queue 

180 JMP CLEAR LABEL 

181 CLEAR LABEL: 

182 

183 ; make DS and ES address 4G of linear memory 

184 MOV CX, LINEAR SEL 

185 MOV DS,CX 

186 MOV ES , CX 

187 

188 ; do board specific initialization 

189 : 

190 F; 

191 rg Eas 

192 ; 

193 

194 

195 ; See Figure 9-5 

196 ; copy EPROM GDT to ram at: 

197 : RAM START + size (STARTUP_DATA) 
198 MOV EAX,RAM START 

199 ADD EAX,OFFSET (end data) 

200 MOV EBX,RAM START 

201 MOV ECX, CS BASE 

202 ADD ECX, OFFSET (GDT_EPROM) 

203 MOV ESI, [ECX] .table linear 

204 MOV EDI, EAX 

205 MOVZX ECX, [ECX].table lim 

206 MOV APP _GDT_ram[EBX] .table_lim,Cx 
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ECX 

EDX, EAX 
APP_GDT_ram[EBX] .table linear, FAX 
EAX , ECX 

BYTE PTR ES: [EDI] ,BYTE PTR DS: [ESI] 


GDT base in descriptor 

ECX , EDX 

[EDX] .bas 0 15+GDT DESC OFF, CX 
ECX,16 

[EDX] .bas 16 23+GDT DESC OFF,CL 
[EDX] .bas 24 31+GDT DESC OFF,CH 


; copy EPROM IDT to ram at: 
; RAM START+size (STARTUP _DATA)+SIZE (EPROM GDT) 


MOV ECX, CS BASE 
ADD ECX, OFFSET (IDT_EPROM) 
MOV ESI, [ECX].table linear 
MOV EDI, EAX 
MOVZX ECX, [ECX].table lim 
MOV APP_IDT_ram[EBX] .table_ lim, CX 
INC ECX 
MOV APP_IDT_ram[EBX] .table linear, EAX 
MOV EBX , EAX 
ADD EAX , ECX 
MOVS BYTE PTR ES: [EDI] ,BYTE PTR DS: [EST] 
; fixup IDT pointer in GDT 
MOV [EDX] .bas_ 0 15+IDT DESC OFF,BX 
ROR EBX, 16 
MOV [EDX] .bas_ 16 23+IDT DESC OFF,BL 
MOV [EDX] .bas 24 31+IDT DESC OFF, BH 
; load GDTR and IDTR 
MOV EBX, RAM START 
DB 66H ; execute a 32 bit LGDT 
LGDT APP _GDT_ ram[EBX] 
DB 66H ; execute a 32 bit LIDT 
LIDT APP_IDT_ ram[EBX] 
; move the TSS 
MOV EDI, EAX 
MOV EBX, TSS INDEX*SIZE (DESC) 
MOV ECX,GDT_DESC_OFF ;build linear address for TSS 
MOV GS,CX 
MOV DH, GS: [EBX] .bas_ 24 31 
MOV DL,GS: [EBX] .bas 16 23 
ROL EDX, 16 
MOV DX,GS: [EBX] .bas_0 15 
MOV ESI, EDX 
LSL ECX, EBX 
INC ECX 
MOV EDX, EAX 
ADD EAX , ECX 
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261 REP MOVS BYTE PTR ES: [EDI] ,BYTE PTR DS: [EST] 

262 

263 ; fixup TSS pointer 

264 MOV GS: [EBX] .bas_0_15,DX 

265 ROL EDX,16 

266 MOV GS: [EBX] .bas_24 31,DH 

267 MOV GS: [EBX] .bas_16 23,DL 

268 ROL EDX,16 

269 ;save start of free ram at linear location RAMSTART 

270 MOV free_mem_linear_base+RAM START, FAX 

271 

272 jassume no LDT used in the initial task - if necessary, 
273 ;code to move the LDT could be added, and should resemble 
274 j;that used to move the TSS 

275 

276 ; load task register 

277 LTR BX ; No task switch, only descriptor loading 
278 ; See Figure 9-6 

279 ; load minimal set of registers necessary to simulate task 
280 j; switch 

281 

282 

283 MOV AX, [EDX] .SS_reg ; start loading registers 

284 MOV EDI, [EDX] .ESP_reg 

285 MOV SS,AX 

286 MOV ESP, EDI ; stack now valid 

287 PUSH DWORD PTR [EDX] .EFLAGS reg 

288 PUSH DWORD PTR [EDX] .CS_reg 

289 PUSH DWORD PTR [EDX] .EIP reg 

290 MOV AX, [EDX] .DS_reg 

291 MOV BX, [EDX] .ES_ reg 

292 MOV DS , AX ; DS and ES no longer linear memory 

293 MOV ES, BX 

294 

295 ; Simulate far jump to initial task 

296 IRETD 

297 


298 STARTUP_CODE ENDS 

*** WARNING #377 IN 298, (PASS 2) SEGMENT CONTAINS PRIVILEGED INSTRUCTION (S) 
299 

300 END STARTUP, DS:STARTUP_DATA, SS:STARTUP_DATA 

301 

302 


ASSEMBLY COMPLETE, 1 WARNING, NO ERRORS. 
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FFFF FFFFH 
START: [CS.BASE+EIP] > FFFF 0000H 
« Jump near start 
* Construct TEMP_GDT 
* LGDT 
* Move to protected mode 
DS, ES = GDT[1] << GB 
_...Base__ 
Limit GDT_SCRATCH 
GDT [1] | Base=0, Limit=4G 
GDT [0] 0 TEMP_GDT 


Figure 9-4. Constructing Temporary GDT and Switching to Protected Mode (Lines 162-172 of List File) 
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FFFF FFFFH 
TSS 
IDT 
GDT 
¢« Move the GDT, IDT, TSS 
from ROM to RAM 
¢ Fix Aliases 
“LTR o < 
Jom TSS RAM 
a IDT RAM 
7 Carney RAM_START 
0 


Figure 9-5. Moving the GDT, IDT, and TSS from ROM to RAM (Lines 196-261 of List File) 
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EIP 
SS = TSS.SS EFLAGS 
ESP = TSS.ESP 
PUSH TSS.EFLAG : 
PUSH TSS.CS 
PUSH TSS.EIP ela 
ES = TSS.ES j 
DS = TSS.DS 
IRET ES 7 > 
cs 
ss 
DS 
GDT 
TSS RAM 
IDT Alias IDT RAM 
— GDT Alias GDT RAM Sas See 
> 0 a 


Figure 9-6. Task Switching (Lines 282-296 of List File) 


9.10.3. MAIN.ASM Source Code 


The file MAIN.ASM shown in Example 9-2 defines the data and stack segments for this application and can be 
substituted with the main module task written in a high-level language that is invoked by the IRET instruction 
executed by STARTUP.ASM. 


Example 9-2. MAIN.ASM 


NAME main_module 
data SEGMENT RW 
dw 1000 dup(?) 
DATA ENDS 
stack stackseg 800 
CODE SEGMENT ER use32 PUBLIC 
main_start: 
nop 
nop 
nop 
CODE ENDS 
END main_start, ds:data, ss:stack 


9.10.4 Supporting Files 


The batch file shown in Example 9-3 can be used to assemble the source code files STARTUP.ASM and MAIN.ASM 
and build the final application. 
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Example 9-3. Batch File to Assemble and Build the Application 


ASM386 STARTUP .ASM 
ASM386 MAIN.ASM 
BLD386 STARTUP.OBJ, MAIN.OBJ buildfile(EPROM.BLD) bootstrap(STARTUP) Bootload 


BLD386 performs several operations in this example: 

It allocates physical memory location to segments and tables. 

It generates tables using the build file and the input files. 

It links object files and resolves references. 

It generates a boot-loadable file to be programmed into the EPROM. 


Example 9-4 shows the build file used as an input to BLD386 to perform the above functions. 


Example 9-4. Build File 


INIT BLD EXAMPLE; 


SEGMENT 
*SEGMENTS (DPL = 0) 
; startup.startup_code(BASE = OFFFF000O0H) 


TASK 
BOOT_TASK(OBJECT = startup, INITIAL,DPL = 0, 
NOT INTENABLED) 
ij PROTECTED MODE TASK(OBJECT = main_module,DPL = 0, 
NOT INTENABLED) 
TABLE 
GDT ( 
LOCATION = GDT_EPROM 
F ENTRY = ( 
10: PROTECTED MODE TASK 
; startup.startup_code 
. startup.startup data 
F main _module.data 
, main_module.code 
1 main_module.stack 
) 
a 
IDT ( 
LOCATION = IDT_ EPROM 
Le 
MEMORY 


( 
RESERVE = (0..3FFFH 
-- Area for the GDT, IDT, TSS copied from ROM 
F 60000H. .OFFFEFFFFH) 


j RANGE = (ROM AREA = ROM (OFFFFOOOOH..OFFFFFFFFH) ) 
-- Eprom size 64K 
r RANGE = (RAM AREA = RAM (4000H..05FFFFH) ) 


9-26 Vol. 3A 


END 


PROCESSOR MANAGEMENT AND INITIALIZATION 


Table 9-6 shows the relationship of each build item with an ASM source file. 


Table 9-6. Relationship Between BLD Item and ASM Source File 


Location of the 
application TSS in 
the GDT 


TSS_INDEX EQU 10 


TABLE GDT( 
ENTRY = (10: PROTECTED_MODE_ 
TASK)) 


Item ASM386 and Startup.A58 BLD386 Controls Effect 
and BLD file 
Bootstrap public startup bootstrap Near jump at OFFFFFFFOH to 
startup: start(startup) Start. 
GDT location public GDT_EPROM TABLE The location of the GDT will be 
GDT_EPROM TABLE_REG <> GDT(location = GDT_EPROM) programmed into the 
GDT_EPROM location. 
IDT location public IDT_EPROM TABLE The location of the IDT will be 
IDT_EPROM TABLE_REG <> IDT(location = IDT_EPROM programmed into the 
IDT_EPROM location. 
RAM start RAM_START equ 400H memory (reserve = (0..3FFFH)) RAM_START is used as the ram 


destination for moving the 
tables. It must be excluded from 
the application’s segment area. 


Put the descriptor of the 
application TSS in GDT entry 10. 


EPROM size and size and location of the initialization | SEGMENT startup.code (base = Initialization code size must be 
location code OFFFFOOOOK) ...memory (RANGE( less than 64K and resides at 
ROM_AREA = ROM(x..y)) upper most 64K of the 4-GByte 
memory space. 
9.11 MICROCODE UPDATE FACILITIES 


The P6 family and later processors have the capability to correct errata by loading an Intel-supplied data block into 
the processor. The data block is called a microcode update. This section describes the mechanisms the BIOS needs 
to provide in order to use this feature during system initialization. It also describes a specification that permits the 
incorporation of future updates into a system BIOS. 


Intel considers the release of a microcode update for a silicon revision to be the equivalent of a processor stepping 
and completes a full-stepping level validation for releases of microcode updates. 


A microcode update is used to correct errata in the processor. The BIOS, which has an update loader, is responsible 
for loading the update on processors during system initialization (Figure 9-7). There are two steps to this process: 
the first is to incorporate the necessary update data blocks into the BIOS; the second is to load update data blocks 


into the processor. 
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Figure 9-7. Applying Microcode Updates 


9.11.1. Microcode Update 


A microcode update consists of an Intel-supplied binary that contains a descriptive header and data. No executable 
code resides within the update. Each microcode update is tailored for a specific list of processor signatures. A 
mismatch of the processor’s signature with the signature contained in the update will result in a failure to load. A 
processor signature includes the extended family, extended model, type, family, model, and stepping of the 
processor (starting with processor family OfH, model 03H, a given microcode update may be associated with one of 
multiple processor signatures; see Section 9.11.2 for details). 


Microcode updates are composed of a multi-byte header, followed by encrypted data and then by an optional 
extended signature table. Table 9-7 provides a definition of the fields; Table 9-8 shows the format of an update. 


The header is 48 bytes. The first 4 bytes of the header contain the header version. The update header and its 
reserved fields are interpreted by software based upon the header version. An encoding scheme guards against 
tampering and provides a means for determining the authenticity of any given update. For microcode updates with 
a data size field equal to OOOOO000H, the size of the microcode update is 2048 bytes. The first 48 bytes contain the 
microcode update header. The remaining 2000 bytes contain encrypted data. 


For microcode updates with a data size not equal to OOOO0000H, the total size field specifies the size of the micro- 
code update. The first 48 bytes contain the microcode update header. The second part of the microcode update is 
the encrypted data. The data size field of the microcode update header specifies the encrypted data size, its value 
must be a multiple of the size of DWORD. The total size field of the microcode update header specifies the 
encrypted data size plus the header size; its value must be in multiples of 1024 bytes (1 KBytes). The optional 
extended signature table if implemented follows the encrypted data, and its size is calculated by (Total Size - (Data 
Size + 48)). 


NOTE 


The optional extended signature table is supported starting with processor family OFH, model 03H. 


Table 9-7. Microcode Update Field Definitions 


Field Name Offset (bytes) Length Description 
(bytes) 
Header Version 0 4 Version number of the update header. 
Update Revision 4 4 Unique version number for the update, the basis for the update 


signature provided by the processor to indicate the current update 
functioning within the processor. Used by the BIOS to authenticate 
the update and verify that the processor loads successfully. The 
value in this field cannot be used for processor stepping identification 
alone. This is a signed 32-bit number. 


Date 8 4 Date of the update creation in binary format: mmddyyyy (e.g. 
07/18/98 is 07181998H). 
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Table 9-7. Microcode Update Field Definitions (Contd.) 


Field Name 


Offset (bytes) 


Length 
(bytes) 


Description 


Processor Signature 


12 


4 


Extended family, extended model, type, family, model, and stepping 
of processor that requires this particular update revision (e.g., 
0Q0000650h). Each microcode update is designed specifically for a 
given extended family, extended model, type, family, model, and 
stepping of the processor. 


Software should use the processor signature field in conjunction with 
the CPUID instruction to determine whether or not an update is 
appropriate to load on a processor. The information encoded within 
this field exactly corresponds to the bit representations returned by 
the CPUID instruction. 


Checksum 


Loader Revision 


20 


Checksum of Update Data and Header. Used to verify the integrity of 
the update header and data. Checksum is correct when the 
summation of all the DWORDs (including the extended Processor 
Signature Table) that comprise the microcode update result in 
OOOOOOOO0H. 


Version number of the loader program needed to correctly load this 
update. The initial version is O0Q0O00001H. 


Processor Flags 


24 


Platform type information is encoded in the lower 8 bits of this 4- 
byte field. Each bit represents a particular platform type for a given 
CPUID. Software should use the processor flags field in conjunction 
with the platform Id bits in MSR (17H) to determine whether or not 
an update is appropriate to load on a processor. Multiple bits may be 
set representing support for multiple platform IDs. 


Data Size 


28 


Specifies the size of the encrypted data in bytes, and must be a 
multiple of DWORDs. If this value is OOOOOOOOH, then the microcode 
update encrypted data is 2000 bytes (or 500 DWORDs). 


Total Size 


32 


Specifies the total size of the microcode update in bytes. It is the 
summation of the header size, the encrypted data size and the size of 
the optional extended signature table. This value is always a multiple 
of 1024. 


Reserved 


36 


12 


Reserved fields for future expansion. 


Update Data 


48 


Data Size or 
2000 


Update data. 


Extended Signature 
Count 


Data Size + 48 


4 


Specifies the number of extended signature structures (Processor 
Signature[n], processor flags[n] and checksum[n]) that exist in this 
microcode update. 


Extended Checksum 


Data Size + 52 


Checksum of update extended processor signature table. Used to 
verify the integrity of the extended processor signature table. 
Checksum is correct when the summation of the DWORDs that 
comprise the extended processor signature table results in 
OOOOOOOO0H. 


Reserved 


Data Size + 56 


Reserved fields. 


Vol. 3A 9-29 


PROCESSOR MANAGEMENT AND INITIALIZATION 


Table 9-7. Microcode Update Field Definitions (Contd.) 


Field Name Offset (bytes) Length Description 
(bytes) 
Processor Signature[n] | DataSize+68+ | 4 Extended family, extended model, type, family, model, and stepping 
(n* 12) of processor that requires this particular update revision (e.g., 


0Q0000650h). Each microcode update is designed specifically for a 
given extended family, extended model, type, family, model, and 
stepping of the processor. 


Software should use the processor signature field in conjunction with 
the CPUID instruction to determine whether or not an update is 
appropriate to load on a processor. The information encoded within 
this field exactly corresponds to the bit representations returned by 
the CPUID instruction. 


Processor Flags[n] Data Size+72+ |4 Platform type information is encoded in the lower 8 bits of this 4- 
(n* 12) byte field. Each bit represents a particular platform type for a given 
CPUID. Software should use the processor flags field in conjunction 
with the platform Id bits in MSR (17H) to determine whether or not 
an update is appropriate to load on a processor. Multiple bits may be 
set representing support for multiple platform IDs. 


Checksum[n] Data Size + 76+ |4 Used by utility software to decompose a microcode update into 
(n* 12) multiple microcode updates where each of the new updates is 
constructed without the optional Extended Processor Signature 
Table. 


To calculate the Checksum, substitute the Primary Processor 
Signature entry and the Processor Flags entry with the 
corresponding Extended Patch entry. Delete the Extended Processor 
Signature Table entries. The Checksum is correct when the 
summation of all DWORDs that comprise the created Extended 
Processor Patch results in OOOOOOOOH. 


Table 9-8. Microcode Update Format 


31 24 16 8 0 Bytes 
Header Version 0 
Update Revision 4 

Month: 8 Day: 8 Year: 16 8 
Processor Signature (CPUID) 12 
wo 7 m a a mn nn 
og) 3) & ™ n - 3 
ne a 
Checksum 16 
Loader Revision 20 
Processor Flags 24 
Reserved (24 bits) S] dG] 8] GB] NN) 3} S 
Data Size 28 
Total Size 32 
Reserved (12 Bytes) 36 
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Table 9-8. Microcode Update Format (Contd.) 


31 24 16 8 0 Bytes 
Update Data (Data Size bytes, or 2000 Bytes if Data Size = O0000000H) 48 
Extended Signature Count ‘n’ Data Size + 

48 
Extended Processor Signature Table Checksum Data Size + 

52 
Reserved (12 Bytes) Data Size + 

56 
Processor Signature[n] Data Size + 

68 + 

(n* 12) 
Processor Flags[n] Data Size + 

72+ 

(n* 12) 
Checksum[n] Data Size + 

76 + 

(n* 12) 


9.11.2 Optional Extended Signature Table 


The extended signature table is a structure that may be appended to the end of the encrypted data when the 
encrypted data only supports a single processor signature (optional case). The extended signature table will always 
be present when the encrypted data supports multiple processor steppings and/or models (required case). 


The extended signature table consists of a 20-byte extended signature header structure, which contains the 
extended signature count, the extended processor signature table checksum, and 12 reserved bytes (Table 9-9). 
Following the extended signature header structure, the extended signature table contains 0-to-n extended 
processor signature structures. 


Each processor signature structure consist of the processor signature, processor flags, and a checksum 
(Table 9-10). 


The extended signature count in the extended signature header structure indicates the number of processor signa- 
ture structures that exist in the extended signature table. 


The extended processor signature table checksum is a checksum of all DWORDs that comprise the extended signa- 
ture table. That includes the extended signature count, extended processor signature table checksum, 12 reserved 
bytes and the n processor signature structures. A valid extended signature table exists when the result of a 
DWORD checksum is OOOOOOOOH. 


Table 9-9. Extended Processor Signature Table Header Structure 


Extended Signature Count ‘n’ Data Size + 48 
Extended Processor Signature Table Checksum Data Size + 52 
Reserved (12 Bytes) Data Size + 56 


Table 9-10. Processor Signature Structure 


Processor Signature[n] Data Size + 68 + (n* 12) 
Processor Flags[n] Data Size + 72 + (n* 12) 
Checksum[n] Data Size + 76 + (n* 12) 
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9.11.3. Processor Identification 


Each microcode update is designed to for a specific processor or set of processors. To determine the correct micro- 
code update to load, software must ensure that one of the processor signatures embedded in the microcode update 
matches the 32-bit processor signature returned by the CPUID instruction when executed by the target processor 
with EAX = 1. Attempting to load a microcode update that does not match a processor signature embedded in the 
microcode update with the processor signature returned by CPUID will cause the BIOS to reject the update. 


Example 9-5 shows how to check for a valid processor signature match between the processor and microcode 
update. 


Example 9-5. Pseudo Code to Validate the Processor Signature 


ProcessorSignature «< CPUID(1) :EAX 


If (Update.HeaderVersion = 00000001h) 
// first check the ProcessorSignature field 
If (ProcessorSignature = Update. ProcessorSignature) 
Success 


// if extended signature is present 
Else If (Update.TotalSize > (Update.DataSize + 48) ) 


{ 
// 


// Assume the Data Size has been used to calculate the 
// location of Update. ProcessorSignature [0]. 


// 


For (N ¢ 0; ((N < Update.ExtendedSignatureCount) AND 
(ProcessorSignature # Update.ProcessorSignature[N])); N++); 


// if the loops ended when the iteration count is 
// less than the number of processor signatures in 
// the table, we have a match 

If (N < Update.ExtendedSignatureCount ) 
Success 

Else 
Fail 


9.11.4 Platform Identification 


In addition to verifying the processor signature, the intended processor platform type must be determined to prop- 
erly target the microcode update. The intended processor platform type is determined by reading the 
IA32_PLATFORM_ID register, (MSR 17H). This 64-bit register must be read using the RDMSR instruction. 


The three platform ID bits, when read as a binary coded decimal (BCD) number, indicate the bit position in the 
microcode update header’s processor flags field associated with the installed processor. The processor flags in the 
48-byte header and the processor flags field associated with the extended processor signature structures may have 
multiple bits set. Each set bit represents a different platform ID that the update supports. 


Register Name: |A32_PLATFORM_ID 
MSR Address: 017H 
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Access: Read Only 


IA32_PLATFORM_ID is a 64-bit register accessed only when referenced as a Qword through a RDMSR instruction. 


Table 9-11. Processor Flags 


Bit Descriptions 

63:53 Reserved 

52:50 Platform Id Bits (RO). The field gives information concerning the intended platform for the processor. See also Table 9-8. 
52 51 50 
O O O- Processor Flag 0 
QO O 1 Processor Flag 1 
O 1  O- Processor Flag 2 
QO 1. 1 Processor Flag 3 
1 0 O- Processor Flag 4 
1 0 1 Processor Flag 5 
1 1. O- Processor Flag 6 
1 1. = 1 Processor Flag 7 

49:0 Reserved 


To validate the platform information, software may implement an algorithm similar to the algorithms in 
Example 9-6. 


Example 9-6. Pseudo Code Example of Processor Flags Test 


Flag < 1 << IA32 PLATFORM ID[52:50] 


If (Update.HeaderVersion = 00000001h) 


{ 


If (Update.ProcessorFlags & Flag) 


{ 
} 


Else 


t 


Load Update 


// 

// Assume the Data Size has been used to calculate the 
// location of Update.ProcessorSignature[N] and a match 
// on Update.ProcessorSignature[N] has already succeeded 


// 
If (Update.ProcessorFlags[n] & Flag) 


Load Update 


9.11.5 Microcode Update Checksum 


Each microcode update contains a DWORD checksum located in the update header. It is software’s responsibility to 
ensure that a microcode update is not corrupt. To check for a corrupt microcode update, software must perform a 
unsigned DWORD (32-bit) checksum of the microcode update. Even though some fields are signed, the checksum 
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procedure treats all DWORDs as unsigned. Microcode updates with a header version equal to O00O00001H must sum 
all DWORDs that comprise the microcode update. A valid checksum check will yield a value of OOOOOO00H. Any 
other value indicates the microcode update is corrupt and should not be loaded. 


The checksum algorithm shown by the pseudo code in Example 9-7 treats the microcode update as an array of 
unsigned DWORDs. If the data size DWORD field at byte offset 32 equals OOOOOOOOH, the size of the encrypted 
data is 2000 bytes, resulting in 500 DWORDs. Otherwise the microcode update size in DWORDs = (Total Size / 4), 
where the total size is a multiple of 1024 bytes (1 KBytes). 


Example 9-7. Pseudo Code Example of Checksum Test 


N < 512 


If (Update.DataSize # 00000000H) 
N ¢ Update.TotalSize / 4 


ChkSum ¢< 0 
For (I ¢< 0; I < N; I++) 


{ 
} 


If (ChkSum = 00000000H) 
Success 

Else 
Fail 


ChkSum < ChkSum + MicrocodeUpdate [T] 


9.11.6 Microcode Update Loader 


This section describes an update loader used to load an update into a P6 family or later processors. It also discusses 
the requirements placed on the BIOS to ensure proper loading. The update loader described contains the minimal 
instructions needed to load an update. The specific instruction sequence that is required to load an update is 
dependent upon the loader revision field contained within the update header. This revision is expected to change 
infrequently (potentially, only when new processor models are introduced). 


Example 9-8 below represents the update loader with a loader revision of OOOO0001H. Note that the microcode 
update must be aligned on a 16-byte boundary and the size of the microcode update must be 1-KByte granular. 


Example 9-8. Assembly Code Example of Simple Microcode Update Loader 


mov ecx,79h ; MSR to write in ECX 

xor eax,eax ; clear EAX 

xor ebx,ebx ; clear EBX 

mov ax,cs ; Segment of microcode update 

shl eax,4 

mov bx,offset Update ; Offset of microcode update 

add eax,ebx ; Linear Address of Update in EAX 

add eax,48d ; Offset of the Update Data within the Update 
xor edx,edx ; Zero in EDX 

WRMSR ; microcode update trigger 


The loader shown in Example 9-8 assumes that update is the address of a microcode update (header and data) 
embedded within the code segment of the BIOS. It also assumes that the processor is operating in real mode. The 
data may reside anywhere in memory, aligned on a 16-byte boundary, that is accessible by the processor within its 
current operating mode. 


Before the BIOS executes the microcode update trigger (WRMSR) instruction, the following must be true: 
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° In 64-bit mode, EAX contains the lower 32-bits of the microcode update linear address. In protected mode, EAX 
contains the full 32-bit linear address of the microcode update. 


° In 64-bit mode, EDX contains the upper 32-bits of the microcode update linear address. In protected mode, 
EDX equals zero. 


¢ ECX contains 79H (address of IA32_BIOS_UPDT_TRIG). 

Other requirements are: 

¢ The addresses for the microcode update data must be in canonical form. 

° If paging is enabled, the microcode update data must map that data as present. 
¢ The microcode update data must start at a 16-byte aligned linear address. 


9.11.6.1 Hard Resets in Update Loading 


The effects of a loaded update are cleared from the processor upon a hard reset. Therefore, each time a hard reset 
is asserted during the BIOS POST, the update must be reloaded on all processors that observed the reset. The 
effects of a loaded update are, however, maintained across a processor INIT. There are no side effects caused by 
loading an update into a processor multiple times. 


9.11.6.2 Update in a Multiprocessor System 


A multiprocessor (MP) system requires loading each processor with update data appropriate for its CPUID and plat- 
form ID bits. The BIOS is responsible for ensuring that this requirement is met and that the loader is located ina 
module executed by all processors in the system. If a system design permits multiple steppings of Pentium 4, Intel 
Xeon, and P6 family processors to exist concurrently; then the BIOS must verify individual processors against the 
update header information to ensure appropriate loading. Given these considerations, it is most practical to load 
the update during MP initialization. 


9.11.6.3_ Update in a System Supporting Intel Hyper-Threading Technology 


Intel Hyper-Threading Technology has implications on the loading of the microcode update. The update must be 
loaded for each core in a physical processor. Thus, for a processor supporting Intel Hyper-Threading Technology, 
only one logical processor per core is required to load the microcode update. Each individual logical processor can 
independently load the update. However, MP initialization must provide some mechanism (e.g. a software sema- 
phore) to force serialization of microcode update loads and to prevent simultaneous load attempts to the same 
core. 


9.11.6.4 Update in a System Supporting Dual-Core Technology 


Dual-core technology has implications on the loading of the microcode update. The microcode update facility is not 
shared between processor cores in the same physical package. The update must be loaded for each core in a phys- 
ical processor. 


If processor core supports Intel Hyper-Threading Technology, the guideline described in Section 9.11.6.3 also 
applies. 


9.11.6.5 Update Loader Enhancements 


The update loader presented in Section 9.11.6, “Microcode Update Loader,” is a minimal implementation that can 
be enhanced to provide additional functionality. Potential enhancements are described below: 


¢ BIOS can incorporate multiple updates to support multiple steppings of the Pentium 4, Intel Xeon, and P6 
family processors. This feature provides for operating in a mixed stepping environment on an MP system and 
enables a user to upgrade to a later version of the processor. In this case, modify the loader to check the CPUID 
and platform ID bits of the processor that it is running on against the available headers before loading a 
particular update. The number of updates is only limited by available BIOS space. 
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¢ A loader can load the update and test the processor to determine if the update was loaded correctly. See 
Section 9.11.7, “Update Signature and Verification.” 


¢ A loader can verify the integrity of the update data by performing a checksum on the double words of the 
update summing to zero. See Section 9.11.5, “Microcode Update Checksum.” 


¢ A loader can provide power-on messages indicating successful loading of an update. 


9.11.7 Update Signature and Verification 


The P6 family and later processors provide capabilities to verify the authenticity of a particular update and to iden- 
tify the current update revision. This section describes the model-specific extensions of processors that support 
this feature. The update verification method below assumes that the BIOS will only verify an update that is more 
recent than the revision currently loaded in the processor. 


CPUID returns a value in a model specific register in addition to its usual register return values. The semantics of 
CPUID cause it to deposit an update ID value in the 64-bit model-specific register at address O8BH 
(IA32_BIOS_SIGN_ID). If no update is present in the processor, the value in the MSR remains unmodified. The 
BIOS must pre-load a zero into the MSR before executing CPUID. If a read of the MSR at 8BH still returns zero after 
executing CPUID, this indicates that no update is present. 


The update ID value returned in the EDX register after RDMSR executes indicates the revision of the update loaded 
in the processor. This value, in combination with the CPUID value returned in the EAX register, uniquely identifies a 
particular update. The signature ID can be directly compared with the update revision field in a microcode update 
header for verification of a correct load. No consecutive updates released for a given stepping of a processor may 
share the same signature. The processor signature returned by CPUID differentiates updates for different step- 
pings. 


9.11.7.1. Determining the Signature 


An update that is successfully loaded into the processor provides a signature that matches the update revision of 
the currently functioning revision. This signature is available any time after the actual update has been loaded. 
Requesting the signature does not have a negative impact upon a loaded update. 


The procedure for determining this signature shown in Example 9-9. 


Example 9-9. Assembly Code to Retrieve the Update Revision 

MOV ECX, O8BH ;IA32_ BIOS SIGN ID 

XOR EAX, EAX ;clear EAX 

XOR EDX, EDX ;clear EDX 

WRMSR ;Load 0 to MSR at 8BH 

MOV EAX, 1 

cpuid 

MOV ECX, O8BH ;1A32_ BIOS SIGN ID 

rdmsr ;Read Model Specific Register 


If there is an update active in the processor, its revision is returned in the EDX register after the RDMSR instruction 
executes. 


IA32_BIOS_SIGN_ID = Microcode Update Signature Register 


MSR Address: O8BH Accessed as a Qword 
Default Value: XXXX XXXX XXXX XXXXh 
Access: Read/Write 


The IA32_BIOS_SIGN_ID register is used to report the microcode update signature when CPUID executes. The 
signature is returned in the upper DWORD (Table 9-12). 
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Table 9-12. Microcode Update Signature 


Bit Description 


63:32 Microcode update signature. This field contains the signature of the currently loaded microcode update when read following 
the execution of the CPUID instruction, function 1. It is required that this register field be pre-loaded with zero prior to 
executing the CPUID, function 1. If the field remains equal to zero, then there is no microcode update loaded. Another non- 
zero value will be the signature. 


31:0 Reserved. 


9.11.7.2 Authenticating the Update 


An update may be authenticated by the BIOS using the signature primitive, described above, and the algorithm in 
Example 9-10. 


Example 9-10. Pseudo Code to Authenticate the Update 


Z << Obtain Update Revision from the Update Header to be authenticated; 
X ¢ Obtain Current Update Signature from MSR 8BH; 


If (Z > X) 


{ 


Load Update that is to be authenticated; 
Y ¢ Obtain New Signature from MSR 8BH; 


If (Z = Y) 
Success 
Else 
Fail 
} 
Else 
Fail 


Example 9-10 requires that the BIOS only authenticate updates that contain a numerically larger revision than the 
currently loaded revision, where Current Signature (X) < New Update Revision (Z). A processor with no loaded 
update is considered to have a revision equal to zero. 


This authentication procedure relies upon the decoding provided by the processor to verify an update from a poten- 
tially hostile source. As an example, this mechanism in conjunction with other safeguards provides security for 
dynamically incorporating field updates into the BIOS. 


9.11.8 Optional Processor Microcode Update Specifications 


This section an interface that an OEM-BIOS may provide to its client system software to manage processor micro- 
code updates. System software may choose to build its own facility to manage microcode updates (e.g. similar to 
the facility described in Section 9.11.6) or rely on a facility provided by the BIOS to perform microcode updates. 


Sections 9.11.8.1-9.11.8.9 describes an extension (Function 0D042H) to the real mode INT 15H service. INT 15H 
0D042H function is one of several alternatives that a BIOS may choose to implement microcode update facility and 
offer to its client application (e.g. an OS). Other alternative microcode update facility that BIOS can choose are 
dependent on platform-specific capabilities, including the Capsule Update mechanism from the UEFI specification 
(www.uefi.org). In this discussion, the application is referred to as the calling program or caller. 


The real mode INT15 call specification described here is an Intel extension to an OEM BIOS. This extension allows 
an application to read and modify the contents of the microcode update data in NVRAM. The update loader, which 
is part of the system BIOS, cannot be updated by the interface. All of the functions defined in the specification must 
be implemented for a system to be considered compliant with the specification. The INT15 functions are accessible 
only from real mode. 
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9.11.8.1 Responsibilities of the BIOS 


If a BIOS passes the presence test (INT 15H, AX = 0D042H, BL = OH), it must implement all of the sub-functions 
defined in the INT 15H, AX = 0D042H specification. There are no optional functions. BIOS must load the appropriate 
update for each processor during system initialization. 


A Header Version of an update block containing the value OFFFFFFFFH indicates that the update block is unused and 
available for storing a new update. 


The BIOS is responsible for providing a region of non-volatile storage (NVRAM) for each potential processor step- 
ping within a system. This storage unit consists of one or more update blocks. An update block is a contiguous 
2048-byte block of memory. The BIOS for a single processor system need only provide update blocks to store one 
microcode update. If the BIOS for a multiple processor system is intended to support mixed processor steppings, 
then the BIOS needs to provide enough update blocks to store each unique microcode update or for each processor 
socket on the OEM’s system board. 


The BIOS is responsible for managing the NVRAM update blocks. This includes garbage collection, such as 
removing microcode updates that exist in NVRAM for which a corresponding processor does not exist in the system. 
This specification only provides the mechanism for ensuring security, the uniqueness of an entry, and that stale 
entries are not loaded. The actual update block management is implementation specific on a per-BIOS basis. 


As an example, the BIOS may use update blocks sequentially in ascending order with CPU signatures sorted versus 
the first available block. In addition, garbage collection may be implemented as a setup option to clear all NVRAM 
slots or as BIOS code that searches and eliminates unused entries during boot. 


NOTES 


For IA-32 processors starting with family OFH and model 03H and Intel 64 processors, the 
microcode update may be as large as 16 KBytes. Thus, BIOS must allocate 8 update blocks for each 
microcode update. In a MP system, a common microcode update may be sufficient for each socket 
in the system. 


For IA-32 processors earlier than family OFH and model 03H, the microcode update is 2 KBytes. An 
MP-capable BIOS that supports multiple steppings must allocate a block for each socket in the 
system. 


A single-processor BIOS that supports variable-sized microcode update and fixed-sized microcode 
update must allocate one 16-KByte region and a second region of at least 2 KBytes. 
The following algorithm (Example 9-11) describes the steps performed during BIOS initialization used to load the 
updates into the processor(s). The algorithm assumes: 


¢ The BIOS ensures that no update contained within NVRAM has a header version or loader version that does not 
match one currently supported by the BIOS. 


¢ The update contains a correct checksum. 
¢ The BIOS ensures that (at most) one update exists for each processor stepping. 
* Older update revisions are not allowed to overwrite more recent ones. 


These requirements are checked by the BIOS during the execution of the write update function of this interface. 
The BIOS sequentially scans through all of the update blocks in NVRAM starting with index 0. The BIOS scans until 
it finds an update where the processor fields in the header match the processor signature (extended family, 
extended model, type, family, model, and stepping) as well as the platform bits of the current processor. 


Example 9-11. Pseudo Code, Checks Required Prior to Loading an Update 


For each processor in the system 


{ 


Determine the Processor Signature via CPUID function 1; 
Determine the Platform Bits ¢ 1 << IA32_ PLATFORM _ID[52:50]; 


For (I © UpdateBlock 0, I < NumOfBlocks; I++) 


{ 


If (Update.Header Version = 00000001H) 


{ 
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If ((Update.ProcessorSignature = Processor Signature) && 
(Update.ProcessorFlags & Platform Bits) ) 


Load Update.UpdateData into the Processor; 
Verify update was correctly loaded into the processor 
Go on to next processor 
Break; 
Else If (Update.TotalSize > (Update.DataSize + 48)) 
N< 0 
While (N < Update.ExtendedSignatureCount) 
If ((Update.ProcessorSignature[N] = 
Processor Signature) && 
(Update.ProcessorFlags[N] & Platform Bits) ) 


Load Update.UpdateData into the Processor; 
Verify update correctly loaded into the processor 
Go on to next processor 

Break; 


} 


N¢Ni 1 
I ¢ I + (Update.TotalSize / 2048) 
If ((Update.TotalSize MOD 2048) = 0) 
IeI+i 


NOTES 


The platform Id bits in IA32_PLATFORM_ID are encoded as a three-bit binary coded decimal field. 
The platform bits in the microcode update header are individually bit encoded. The algorithm must 
do a translation from one format to the other prior to doing a check. 


When performing the INT 15H, 0D042H functions, the BIOS must assume that the caller has no knowledge of plat- 
form specific requirements. It is the responsibility of BIOS calls to manage all chipset and platform specific prereq- 
uisites for managing the NVRAM device. When writing the update data using the Write Update sub-function, the 
BIOS must maintain implementation specific data requirements (such as the update of NVRAM checksum). The 
BIOS should also attempt to verify the success of write operations on the storage device used to record the update. 


9.11.8.2 Responsibilities of the Calling Program 


This section of the document lists the responsibilities of a calling program using the interface specifications to load 
microcode update(s) into BIOS NVRAM. 


¢ The calling program should call the INT 15H, 0D042H functions from a pure real mode program and should be 
executing on a system that is running in pure real mode. 


The caller should issue the presence test function (sub function 0) and verify the signature and return codes of 
that function. 


¢  Itis important that the calling program provides the required scratch RAM buffers for the BIOS and the proper 
stack size as specified in the interface definition. 


¢ The calling program should read any update data that already exists in the BIOS in order to make decisions 
about the appropriateness of loading the update. The BIOS must refuse to overwrite a newer update with an 
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older version. The update header contains information about version and processor specifics for the calling 
program to make an intelligent decision about loading. 


¢ There can be no ambiguous updates. The BIOS must refuse to allow multiple updates for the same CPU to exist 
at the same time; it also must refuse to load updates for processors that don’t exist on the system. 


¢ The calling application should implement a verify function that is run after the update write function success- 
fully completes. This function reads back the update and verifies that the BIOS returned an image identical to 
the one that was written. 


Example 9-12 represents a calling program. 


Example 9-12. INT 15 D042 Calling Program Pseudo-code 


// We must be in real mode 
If the system is not in Real mode exit 


// Detect presence of Genuine Intel processor(s) that can be updated 
// using (CPUID) 


If no Intel processors exist that can be updated exit 

// Detect the presence of the Intel microcode update extensions 

If the BIOS fails the PresenceTestexit 

// If the APIC is enabled, see if any other processors are out there 


Read IA32 APICBASE 
If APIC enabled 


Send Broadcast Message to all processors except self via APIC 
Have all processors execute CPUID, record the Processor Signature 
(i.e.,Extended Family, Extended Model, Type, Family, Model, Stepping) 
Have all processors read IA32 PLATFORM ID[52:50], record Platform 
Id Bits 
If current processor cannot be updated 
exit 
// 
// Determine the number of unique update blocks needed for this system 
// 


NumBlocks = 0 
For each processor 


{ 
If ((this is a unique processor stepping) AND 
(we have a unique update in the database for this processor) ) 
{ 
Checksum the update from the database; 
If Checksum fails 
exit 
NumBlocks < NumBlocks + size of microcode update / 2048 
} 
} 
// 
// Do we have enough update slots for all CPUs? 
// 
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If there are more blocks required to support the unique processor steppings than update blocks 
provided by the BIOS exit 


// 
// Do we need any update blocks at all? If not, we are done 
// 
If (NumBlocks = 0) 
exit 
// 
// Record updates for processors in NVRAM. 
// 
For (I=0; I<NumBlocks; I++) 
{ 
// 
// “oad each Update 
// 


Issue the WriteUpdate function 


If (STORAGE FULL) returned 


{ 


Display Error -- BIOS is not managing NVRAM appropriately 
exit 


If (INVALID REVISION) returned 


Display Message: More recent update already loaded in NVRAM for 
this stepping 
continue 


If any other error returned 


Display Diagnostic 
exit 


} 


// 
// Verify the update was loaded correctly 


// 


Issue the ReadUpdate function 


If an error occurred 


Display Diagnostic 
exit 


// Compare the Update read to that written 


If (Update read # Update written) 


Display Diagnostic 
exit 


} 


I ¢ I+ (size of microcode update / 2048) 


} 
// 


// Enable Update Loading, and inform user 
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// 


Issue the Update Control function with Task = Enable. 


9.11.8.3 Microcode Update Functions 


Table 9-13 defines the processor microcode update functions that implementations of INT 15H 0D042H must 
support. 


Table 9-13. Microcode Update Functions 


Microcode Update Function | Function Description Required/Optional 
Number 

Presence test OOH Returns information about the supported functions. Required 

Write update data 01H Writes one of the update data areas (slots). Required 

Update control 02H Globally controls the loading of updates. Required 

Read update data 03H Reads one of the update data areas (slots). Required 


9.11.8.4 INT 15H-based Interface 


If an OEM-BIOS is implementing INT 15H OD042H interface and offer to its client, the BIOS should allow additional 
microcode updates to be added to system flash. 


The program that calls this interface is responsible for providing three 64-kilobyte RAM areas for BIOS use during 
calls to the read and write functions. These RAM scratch pads can be used by the BIOS for any purpose, but only 
for the duration of the function call. The calling routine places real mode segments pointing to the RAM blocks in 
the CX, DX and SI registers. Calls to functions in this interface must be made with a minimum of 32 kilobytes of 
stack available to the BIOS. 


In general, each function returns with CF cleared and AH contains the returned status. The general return codes 
and other constant definitions are listed in Section 9.11.8.9, “Return Codes.” 


The OEM error field (AL) is provided for the OEM to return additional error information specific to the platform. If 
the BIOS provides no additional information about the error, OEM error must be set to SUCCESS. The OEM error 
field is undefined if AH contains either SUCCESS (00H) or NOT_IMPLEMENTED (86H). In all other cases, it must be 
set with either SUCCESS or a value meaningful to the OEM. 


The following sections describe functions provided by the INT15H-based interface. 


9.11.8.5 Function OOH—Presence Test 


This function verifies that the BIOS has implemented required microcode update functions. Table 9-14 lists the 
parameters and return codes for the function. 


Table 9-14. Parameters for the Presence Test 


Input 

AX Function Code 0D042H 

BL Sub-function OOH - Presence test 

Output 

CF Carry Flag Carry Set - Failure - AH contains status 
Carry Clear - All return values valid 

AH Return Code 

AL OEM Error Additional OEM information. 

EBX Signature Part 1 ‘INTE’ - Part one of the signature 

ECX Signature Part 2 ‘LPEP’- Part two of the signature 

EDX Loader Version Version number of the microcode update loader 
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Table 9-14. Parameters for the Presence Test (Contd.) 


Input 


Sl 


Update Count 


Number of 2048 update blocks in NVRAM the BIOS allocated to storing 
microcode updates 


Return Codes (see Table 9-19 for code definitions 


SUCCESS 


The function completed successfully. 


NOT_IMPLEMENTED 


The function is not implemented. 


In order to assure that the BIOS function is present, the caller must verify the carry flag, the return code, and the 
64-bit signature. The update count reflects the number of 2048-byte blocks available for storage within one non- 


volatile RAM. 


The loader version number refers to the revision of the update loader program that is included in the system BIOS 


image. 


9.11.8.6 Function 01H—Write Microcode Update Data 


This function integrates a new microcode update into the BIOS storage device. Table 9-15 lists the parameters and 


return codes for the function. 


Table 9-15. Parameters for the Write Update Data Function 


Input 

AX Function Code 0D042H 

BL Sub-function 01H - Write update 

ES:DI Update Address Real Mode pointer to the Intel Update structure. This buffer is 2048 bytes in 
length if the processor supports only fixed-size microcode update or... 
Real Mode pointer to the Intel Update structure. This buffer is 64 KBytes in 
length if the processor supports a variable-size microcode update. 

CX Scratch Pad1 Real mode segment address of 64 KBytes of RAM block 

DX Scratch Pad2 Real mode segment address of 64 KBytes of RAM block 

Sl Scratch Pad3 Real mode segment address of 64 KBytes of RAM block 

SS:SP Stack pointer 32 KBytes of stack minimum 

Output 

CF Carry Flag Carry Set - Failure - AH Contains status 
Carry Clear - All return values valid 

AH Return Code Status of the call 

AL OEM Error Additional OEM information 


Return Codes (see Table 9-19 for code definitions 


SUCCESS 


The function completed successfully. 


NOT_IMPLEMENTED 


The function is not implemented. 


WRITE_FAILURE 
ERASE_FAILURE 


A failure occurred because of the inability to write the storage device. 


A failure occurred because of the inability to erase the storage device. 


READ_FAILURE 


A failure occurred because of the inability to read the storage device. 
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Table 9-15. Parameters for the Write Update Data Function (Contd.) 


Input 

STORAGE_FULL The BIOS non-volatile storage area is unable to accommodate the update 
because all available update blocks are filled with updates that are needed for 
processors in the system. 

CPU_NOT_PRESENT The processor stepping does not currently exist in the system. 

INVALID_HEADER The update header contains a header or loader version that is not recognized by 
the BIOS. 

INVALID_HEADER_CS The update does not checksum correctly. 

SECURITY_FAILURE The processor rejected the update. 

INVALID_REVISION The same or more recent revision of the update exists in the storage device. 

Description 


The BIOS is responsible for selecting an appropriate update block in the non-volatile storage for storing the new 
update. This BIOS is also responsible for ensuring the integrity of the information provided by the caller, including 
authenticating the proposed update before incorporating it into storage. 


Before writing the update block into NVRAM, the BIOS should ensure that the update structure meets the following 
criteria in the following order: 


1. The update header version should be equal to an update header version recognized by the BIOS. 


2. The update loader version in the update header should be equal to the update loader version contained within 
the BIOS image. 


3. The update block must checksum. This checksum is computed as a 32-bit summation of all double words in the 
structure, including the header, data, and processor signature table. 


The BIOS selects update block(s) in non-volatile storage for storing the candidate update. The BIOS can select any 
available update block as long as it guarantees that only a single update exists for any given processor stepping in 
non-volatile storage. If the update block selected already contains an update, the following additional criteria apply 
to overwrite it: 


¢ The processor signature in the proposed update must be equal to the processor signature in the header of the 
current update in NVRAM (Processor Signature + platform ID bits). 


¢ The update revision in the proposed update should be greater than the update revision in the header of the 
current update in NVRAM. 


If no unused update blocks are available and the above criteria are not met, the BIOS can overwrite update 
block(s) for a processor stepping that is no longer present in the system. This can be done by scanning the update 
blocks and comparing the processor steppings, identified in the MP Specification table, to the processor steppings 
that currently exist in the system. 


Finally, before storing the proposed update in NVRAM, the BIOS must verify the authenticity of the update via the 
mechanism described in Section 9.11.6, “Microcode Update Loader.” This includes loading the update into the 
current processor, executing the CPUID instruction, reading MSR O8Bh, and comparing a calculated value with the 
update revision in the proposed update header for equality. 


When performing the write update function, the BIOS must record the entire update, including the header, the 
update data, and the extended processor signature table (if applicable). When writing an update, the original 
contents may be overwritten, assuming the above criteria have been met. It is the responsibility of the BIOS to 
ensure that more recent updates are not overwritten through the use of this BIOS call, and that only a single 
update exists within the NVRAM for any processor stepping and platform ID. 


Figure 9-8 and Figure 9-9 show the process the BIOS follows to choose an update block and ensure the integrity of 
the data when it stores the new microcode update. 
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Figure 9-8. Microcode Update Write Operation Flow [1] 
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Figure 9-9. Microcode Update Write Operation Flow [2] 


9.11.8.7. Function O2H—Microcode Update Control 


This function enables loading of binary updates into the processor. Table 9-16 lists the parameters and return codes 
for the function. 
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Table 9-16. Parameters for the Control Update Sub-function 


Input 
AX Function Code 0D042H 
BL Sub-function 02H - Control update 
BH Task See the description below. 
CX Scratch Pad1 Real mode segment of 64 KBytes of RAM block 
DX Scratch Pad2 Real mode segment of 64 KBytes of RAM block 
SI Scratch Pad3 Real mode segment of 64 KBytes of RAM block 
SS:SP Stack pointer 32 kilobytes of stack minimum 
Output 
CF Carry Flag Carry Set - Failure - AH contains status 
Carry Clear - All return values valid. 
AH Return Code Status of the call 
AL OEM Error Additional OEM Information. 
BL Update Status Either enable or disable indicator 
Return Codes (see Table 9-19 for code definitions) 
SUCCESS Function completed successfully. 
READ_FAILURE A failure occurred because of the inability to read the storage device. 


This control is provided on a global basis for all updates and processors. The caller can determine the current status 
of update loading (enabled or disabled) without changing the state. The function does not allow the caller to disable 
loading of binary updates, as this poses a security risk. 


The caller specifies the requested operation by placing one of the values from Table 9-17 in the BH register. After 
successfully completing this function, the BL register contains either the enable or the disable designator. Note that 
if the function fails, the update status return value is undefined. 


Table 9-17. Mnemonic Values 


Mnemonic Value Meaning 
Enable 1 Enable the Update loading at initialization time. 
Query 2 Determine the current state of the update control without changing its status. 


The READ_FAILURE error code returned by this function has meaning only if the control function is implemented in 
the BIOS NVRAM. The state of this feature (enabled/disabled) can also be implemented using CMOS RAM bits 
where READ failure errors cannot occur. 


9.11.8.8 Function O3H—Read Microcode Update Data 


This function reads a currently installed microcode update from the BIOS storage into a caller-provided RAM buffer. 
Table 9-18 lists the parameters and return codes. 


Table 9-18. Parameters for the Read Microcode Update Data Function 


Input 

AX Function Code 0D042H 

BL Sub-function 03H - Read Update 

ES:DI Buffer Address Real Mode pointer to the Intel Update structure that 
will be written with the binary data 
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Table 9-18. Parameters for the Read Microcode Update Data Function (Contd.) 


ECX Scratch Pad1 Real Mode Segment address of 64 KBytes of RAM 
Block (lower 16 bits) 

ECX Scratch Pad2 Real Mode Segment address of 64 KBytes of RAM 
Block (upper 16 bits) 

DX Scratch Pad3 Real Mode Segment address of 64 KBytes of RAM 
Block 

SS:SP Stack pointer 32 KBytes of Stack Minimum 

SI Update Number This is the index number of the update block to be 


read. This value is zero based and must be less than 
the update count returned from the presence test 


function. 

Output 

CF Carry Flag Carry Set - Failure - AH contains Status 

Carry Clear - All return values are 

valid. 

AH Return Code Status of the Call 

AL OEM Error Additional OEM Information 

Return Codes (see Table 9-19 for code definitions) 

SUCCESS The function completed successfully. 

READ_FAILURE There was a failure because of the inability to read the 
storage device. 

UPDATE_NUM_INVALID Update number exceeds the maximum number of 
update blocks implemented by the BIOS. 

NOT_EMPTY The specified update block is a subsequent block in use 
to store a valid microcode update that spans multiple 
blocks. 

The specified block is not a header block and is not 
empty. 


The read function enables the caller to read any microcode update data that already exists in a BIOS and make 
decisions about the addition of new updates. As a result of a successful call, the BIOS copies the microcode update 
into the location pointed to by ES:DI, with the contents of all Update block(s) that are used to store the specified 
microcode update. 


If the specified block is not a header block, but does contain valid data from a microcode update that spans multiple 
update blocks, then the BIOS must return Failure with the NOT_EMPTY error code in AH. 


An update block is considered unused and available for storing a new update if its Header Version contains the 
value OFFFFFFFFH after return from this function call. The actual implementation of NVRAM storage management 
is not specified here and is BIOS dependent. As an example, the actual data value used to represent an empty 
block by the BIOS may be zero, rather than OFFFFFFFFH. The BIOS is responsible for translating this information 
into the header provided by this function. 


9.11.8.9 Return Codes 


After the call has been made, the return codes listed in Table 9-19 are available in the AH register. 
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Table 9-19. Return Code Definitions 


Return Code Value Description 

SUCCESS OOH The function completed successfully. 

NOT_IMPLEMENTED 86H The function is not implemented. 

ERASE_FAILURE 90H A failure because of the inability to erase the storage device. 
WRITE_FAILURE 91H A failure because of the inability to write the storage device. 
READ_FAILURE 92H A failure because of the inability to read the storage device. 
STORAGE_FULL 93H The BIOS non-volatile storage area is unable to accommodate the update 


because all available update blocks are filled with updates that are needed 
for processors in the system. 


CPU_NOT_PRESENT 94H The processor stepping does not currently exist in the system. 

INVALID_HEADER 95H The update header contains a header or loader version that is not 
recognized by the BIOS. 

INVALID_HEADER_CS 96H The update does not checksum correctly. 

SECURITY_FAILURE 97H The update was rejected by the processor. 

INVALID_REVISION 98H The same or more recent revision of the update exists in the storage device. 

UPDATE_NUM_INVALID 99H The update number exceeds the maximum number of update blocks 
implemented by the BIOS. 

NOT_EMPTY 9AH The specified update block is a subsequent block in use to store a valid 


microcode update that spans multiple blocks. 


The specified block is not a header block and is not empty. 
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CHAPTER 10 
ADVANCED PROGRAMMABLE 
INTERRUPT CONTROLLER (APIC) 


The Advanced Programmable Interrupt Controller (APIC), referred to in the following sections as the local APIC, 
was introduced into the IA-32 processors with the Pentium processor (see Section 21.27, “Advanced Program- 
mable Interrupt Controller (APIC)”) and is included in the P6 family, Pentium 4, Intel Xeon processors, and other 
more recent Intel 64 and IA-32 processor families (see Section 10.4.2, “Presence of the Local APIC”). The local 
APIC performs two primary functions for the processor: 


¢ It receives interrupts from the processor’s interrupt pins, from internal sources and from an external I/O APIC 
(or other external interrupt controller). It sends these to the processor core for handling. 


¢ In multiple processor (MP) systems, it sends and receives interprocessor interrupt (IPI) messages to and from 
other logical processors on the system bus. IPI messages can be used to distribute interrupts among the 
processors in the system or to execute system wide functions (such as, booting up processors or distributing 
work among a group of processors). 


The external I/O APIC is part of Intel’s system chip set. Its primary function is to receive external interrupt events 
from the system and its associated I/O devices and relay them to the local APIC as interrupt messages. In MP 
systems, the I/O APIC also provides a mechanism for distributing external interrupts to the local APICs of selected 
processors or groups of processors on the system bus. 


This chapter provides a description of the local APIC and its programming interface. It also provides an overview of 
the interface between the local APIC and the I/O APIC. Contact Intel for detailed information about the I/O APIC. 


When a local APIC has sent an interrupt to its processor core for handling, the processor uses the interrupt and 
exception handling mechanism described in Chapter 6, “Interrupt and Exception Handling.” See Section 6.1, “Inter- 
rupt and Exception Overview,” for an introduction to interrupt and exception handling. 


10.1 LOCAL AND I/0 APIC OVERVIEW 


Each local APIC consists of a set of APIC registers (see Table 10-1) and associated hardware that control the 
delivery of interrupts to the processor core and the generation of IPI messages. The APIC registers are memory 
mapped and can be read and written to using the MOV instruction. 


Local APICs can receive interrupts from the following sources: 


¢ Locally connected I/O devices — These interrupts originate as an edge or level asserted by an I/O device 
that is connected directly to the processor’s local interrupt pins (LINTO and LINT1). The I/O devices may also 
be connected to an 8259-type interrupt controller that is in turn connected to the processor through one of the 
local interrupt pins. 


° Externally connected I/O devices — These interrupts originate as an edge or level asserted by an I/O 
device that is connected to the interrupt input pins of an I/O APIC. Interrupts are sent as I/O interrupt 
messages from the I/O APIC to one or more of the processors in the system. 


¢ Inter-processor interrupts (IPIs) — An Intel 64 or IA-32 processor can use the IPI mechanism to interrupt 
another processor or group of processors on the system bus. IPIs are used for software self-interrupts, 
interrupt forwarding, or preemptive scheduling. 


¢ APIC timer generated interrupts — The local APIC timer can be programmed to send a local interrupt to its 
associated processor when a programmed count is reached (see Section 10.5.4, “APIC Timer”). 


¢ Performance monitoring counter interrupts — P6 family, Pentium 4, and Intel Xeon processors provide the 
ability to send an interrupt to its associated processor when a performance-monitoring counter overflows (see 
Section 18.6.3.5.8, “Generating an Interrupt on Overflow”). 


¢ Thermal Sensor interrupts — Pentium 4 and Intel Xeon processors provide the ability to send an interrupt to 
themselves when the internal thermal sensor has been tripped (see Section 14.8.2, “Thermal Monitor”). 
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¢ APIC internal error interrupts — When an error condition is recognized within the local APIC (such as an 
attempt to access an unimplemented register), the APIC can be programmed to send an interrupt to its 
associated processor (see Section 10.5.3, “Error Handling”). 


Of these interrupt sources: the processor’s LINTO and LINT1 pins, the APIC timer, the performance-monitoring 
counters, the thermal sensor, and the internal APIC error detector are referred to as local interrupt sources. 
Upon receiving a signal from a local interrupt source, the local APIC delivers the interrupt to the processor core 
using an interrupt delivery protocol that has been set up through a group of APIC registers called the local vector 
table or LVT (see Section 10.5.1, “Local Vector Table”). A separate entry is provided in the local vector table for 
each local interrupt source, which allows a specific interrupt delivery protocol to be set up for each source. For 
example, if the LINT1 pin is going to be used as an NMI pin, the LINT1 entry in the local vector table can be set up 
to deliver an interrupt with vector number 2 (NMI interrupt) to the processor core. 


The local APIC handles interrupts from the other two interrupt sources (externally connected I/O devices and IPIs) 
through its IPI message handling facilities. 


A processor can generate IPIs by programming the interrupt command register (ICR) in its local APIC (see Section 
10.6.1, “Interrupt Command Register (ICR)”). The act of writing to the ICR causes an IPI message to be generated 
and issued on the system bus (for Pentium 4 and Intel Xeon processors) or on the APIC bus (for Pentium and P6 
family processors). See Section 10.2, “System Bus Vs. APIC Bus.” 


IPIs can be sent to other processors in the system or to the originating processor (self-interrupts). When the target 
processor receives an IPI message, its local APIC handles the message automatically (using information included 
in the message such as vector number and trigger mode). See Section 10.6, “Issuing Interprocessor Interrupts,” 
for a detailed explanation of the local APIC’s IPI message delivery and acceptance mechanism. 


The local APIC can also receive interrupts from externally connected devices through the I/O APIC (see 
Figure 10-1). The I/O APIC is responsible for receiving interrupts generated by system hardware and I/O devices 
and forwarding them to the local APIC as interrupt messages. 


Pentium 4 and Pentium and P6 
Intel Xeon Processors Family Processors 
Processor Core Processor Core 
Local APIC Local APIC 
Interrupt Local Interrupt Local 
Messages Interrupts Messages Interrupts 


Interrupt} System Bus 3-Wire APIC Bus 
Messages 
Bridge <—— External 
Leal }<<—— Interrupts 


ad System Chip Set 


<—— External 
eo ene |}<<—;— Interrupts 


System Chip Set 


Figure 10-1. Relationship of Local APIC and I/O APIC In Single-Processor Systems 


Individual pins on the I/O APIC can be programmed to generate a specific interrupt vector when asserted. The I/O 
APIC also has a “virtual wire mode” that allows it to communicate with a standard 8259A-style external interrupt 
controller. Note that the local APIC can be disabled (see Section 10.4.3, “Enabling or Disabling the Local APIC”). 
This allows an associated processor core to receive interrupts directly from an 8259A interrupt controller. 
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Both the local APIC and the I/O APIC are designed to operate in MP systems (see Figures 10-2 and 10-3). Each 
local APIC handles interrupts from the I/O APIC, IPIs from processors on the system bus, and self-generated inter- 


rupts. Interrupts can also be delivered to the individual processors through the local interrupt pins; however, this 
mechanism is commonly not used in MP systems. 


Processor #1 Processor #2 Processor #3 Processor #3 
CPU CPU CPU CPU 
Local APIC Local APIC Local APIC Local APIC 
Interrupt Interrupt Interrupt Interrupt 
Messages | IPIs Messaues IPIs Messages IPIS_ Messages IPIs 
Interrupt Processor System Bus 
Messages 
Bridge 


<—— External 
WOPOIS ~<<—— Interrupts 


System Chip Set 


Figure 10-2. Local APICs and I/O APIC When Intel Xeon Processors Are Used in Multiple-Processor Systems 


rrocessor #1 Processor #2 Processor #3 Processor #4 
CPU CPU CPU CPU 
Local APIC Local APIC Local APIC Local APIC 
Interrupt IPIs Interrupt IPIs Interrupt IPIs Interrupt IPIs 
Messages Messages Messages Messages 
Interrupt 3-wire APIC Bus 
Messages 


External -——> | 
Interrupts || VO APIC 


System Chip Set 


Figure 10-3. Local APICs and I/O APIC When P6 Family Processors Are Used in Multiple-Processor Systems 


The IPI mechanism is typically used in MP systems to send fixed interrupts (interrupts for a specific vector number) 
and special-purpose interrupts to processors on the system bus. For example, a local APIC can use an IPI to 
forward a fixed interrupt to another processor for servicing. Special-purpose IPIs (including NMI, INIT, SMI and 
SIPI IPIs) allow one or more processors on the system bus to perform system-wide boot-up and control functions. 


The following sections focus on the local APIC and its implementation in the Pentium 4, Intel Xeon, and P6 family 
processors. In these sections, the terms “local APIC” and “I/O APIC” refer to local and I/O APICs used with the P6 
family processors and to local and I/O xAPICs used with the Pentium 4 and Intel Xeon processors (see Section 
10.3, “The Intel® 82489DxX External APIC, the APIC, the xAPIC, and the X2APIC”). 
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10.2 SYSTEM BUS VS. APIC BUS 


For the P6 family and Pentium processors, the I/O APIC and local APICs communicate through the 3-wire inter- 
APIC bus (see Figure 10-3). Local APICs also use the APIC bus to send and receive IPIs. The APIC bus and its 
messages are invisible to software and are not classed as architectural. 


Beginning with the Pentium 4 and Intel Xeon processors, the I/O APIC and local APICs (using the xAPIC architec- 
ture) communicate through the system bus (see Figure 10-2). The I/O APIC sends interrupt requests to the 
processors on the system bus through bridge hardware that is part of the Intel chip set. The bridge hardware 
generates the interrupt messages that go to the local APICs. IPIs between local APICs are transmitted directly on 
the system bus. 


10.3. THE INTEL® 82489DX EXTERNAL APIC, THE APIC, THE XAPIC, AND THE 
X2APIC 


The local APIC in the P6 family and Pentium processors is an architectural subset of the Intel® 82489DxX external 
APIC. See Section 21.27.1, “Software Visible Differences Between the Local APIC and the 82489Dx.” 


The APIC architecture used in the Pentium 4 and Intel Xeon processors (called the xAPIC architecture) is an exten- 
sion of the APIC architecture found in the P6 family processors. The primary difference between the APIC and 
xAPIC architectures is that with the xAPIC architecture, the local APICs and the I/O APIC communicate through the 
system bus. With the APIC architecture, they communication through the APIC bus (see Section 10.2, “System Bus 
Vs. APIC Bus”). Also, some APIC architectural features have been extended and/or modified in the xAPIC architec- 
ture. These extensions and modifications are described in Section 10.4 through Section 10.10. 


The basic operating mode of the xAPIC is xAPIC mode. The x2APIC architecture is an extension of the xAPIC 
architecture, primarily to increase processor addressability. The x2APIC architecture provides backward compati- 
bility to the xAPIC architecture and forward extendability for future Intel platform innovations. These extensions 
and modifications are supported by a new mode of execution (x2APIC mode) are detailed in Section 10.12. 


10.4 LOCAL APIC 


The following sections describe the architecture of the local APIC and how to detect it, identify it, and determine its 
status. Descriptions of how to program the local APIC are given in Section 10.5.1, “Local Vector Table,” and Section 
10.6.1, “Interrupt Command Register (ICR).” 


10.4.1. The Local APIC Block Diagram 


Figure 10-4 gives a functional block diagram for the local APIC. Software interacts with the local APIC by reading 
and writing its registers. APIC registers are memory-mapped to a 4-KByte region of the processor’s physical 
address space with an initial starting address of FEEQOOOOH. For correct APIC operation, this address space must 
be mapped to an area of memory that has been designated as strong uncacheable (UC). See Section 11.3, 
“Methods of Caching Available.” 


In MP system configurations, the APIC registers for Intel 64 or IA-32 processors on the system bus are initially 
mapped to the same 4-KByte region of the physical address space. Software has the option of changing initial 
mapping to a different 4-KByte region for all the local APICs or of mapping the APIC registers for each local APIC to 
its own 4-KByte region. Section 10.4.5, “Relocating the Local APIC Registers,” describes how to relocate the base 
address for APIC registers. 


On processors supporting x2APIC architecture (indicated by CPUID.O1H:ECX[21] = 1), the local APIC supports 
operation both in xAPIC mode and (if enabled by software) in x2APIC mode. x2APIC mode provides extended 
processor addressability (see Section 10.12). 
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NOTE 


For P6 family, Pentium 4, and Intel Xeon processors, the APIC handles all memory accesses to 
addresses within the 4-KByte APIC register space internally and no external bus cycles are 
produced. For the Pentium processors with an on-chip APIC, bus cycles are produced for accesses 
to the APIC register space. Thus, for software intended to run on Pentium processors, system 
software should explicitly not map the APIC register space to regular system memory. Doing so can 
result in an invalid opcode exception (#UD) being generated or unpredictable execution. 
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1. Introduced in P6 family processors. 

2. Introduced in the Pentium 4 and Intel Xeon processors. 

3. Three-wire APIC bus in P6 family and Pentium processors. 
4. Not implemented in Pentium 4 and Intel Xeon processors. 


Figure 10-4. Local APIC Structure 


Table 10-1 shows how the APIC registers are mapped into the 4-KByte APIC register space. Registers are 32 bits, 
64 bits, or 256 bits in width; all are aligned on 128-bit boundaries. All 32-bit registers should be accessed using 
128-bit aligned 32-bit loads or stores. Some processors may support loads and stores of less than 32 bits to some 
of the APIC registers. This is model specific behavior and is not guaranteed to work on all processors. Any 
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FP/MMX/SSE access to an APIC register, or any access that touches bytes 4 through 15 of an APIC register may 
cause undefined behavior and must not be executed. This undefined behavior could include hangs, incorrect results 
or unexpected exceptions, including machine checks, and may vary between implementations. Wider registers 
(64-bit or 256-bit) must be accessed using multiple 32-bit loads or stores, with all accesses being 128-bit aligned. 


The local APIC registers listed in Table 10-1 are not MSRs. The only MSR associated with the programming of the 
local APIC is the IA32_APIC_BASE MSR (see Section 10.4.3, “Enabling or Disabling the Local APIC”). 


NOTE 


In processors based on Intel microarchitecture code name Nehalem! the Local APIC ID Register is 
no longer Read/Write; it is Read Only. 


Table 10-1 Local APIC Register Address Map 


Address Register Name Software Read/Write 

FEEO OOOOH Reserved 

FEEO 0010H Reserved 

FEEO 0020H Local APIC ID Register Read/Write. 

FEEO 0030H Local APIC Version Register Read Only. 

FEEO 0040H Reserved 

FEEO OO50H Reserved 

FEEO OO60H Reserved 

FEEO 0070H Reserved 

FEEO OO80H Task Priority Register (TPR) Read/Write. 

FEEO OOS0H Arbitration Priority Register! (APR) Read Only. 

FEEO OOAOH Processor Priority Register (PPR) Read Only. 

FEEO OOBOH EOI Register Write Only. 

FEEO OOCOH Remote Read Register! (RRD) Read Only 

FEEO OODOH Logical Destination Register Read/Write. 

FEEO OOEOH Destination Format Register Read/Write (See Section 
10.6.2.2). 

FEEO OOFOH Spurious Interrupt Vector Register Read/Write (see Section 10.9. 

FEEO 0100H In-Service Register (ISR); bits 31:0 Read Only. 

FEEO 0110H In-Service Register (ISR); bits 63:32 Read Only. 

FEEO 0120H In-Service Register (ISR); bits 95:64 Read Only. 

FEEO 0130H In-Service Register (ISR); bits 127:96 Read Only. 

FEEO 0140H In-Service Register (ISR); bits 159:128 Read Only. 

FEEO 0150H In-Service Register (ISR); bits 191:160 Read Only. 

FEEO 0160H In-Service Register (ISR); bits 223:192 Read Only. 

FEEO 0170H In-Service Register (ISR); bits 255:224 Read Only. 

FEEO 0180H Trigger Mode Register (TMR); bits 31:0 Read Only. 

FEEO 0190H Trigger Mode Register (TMR); bits 63:32 Read Only. 

FEEO 01A0H Trigger Mode Register (TMR); bits 95:64 Read Only. 


1. See Table 2-1, “CPUID Signature Values of DisplayFamily_DisplayModel,” on page 1, and Section 2.8, “MSRs In the Intel® Microarchi- 
tecture Code Name Nehalem” of the Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 4 to determine which 
processors are based on Nehalem microarchitecture. 
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Table 10-1 Local APIC Register Address Map (Contd.) 


Address Register Name Software Read/Write 
FEEO 01BOH Trigger Mode Register (TMR); bits 127:96 Read Only. 
FEEO 01COH Trigger Mode Register (TMR); bits 159:128 Read Only. 
FEEO 01D0H Trigger Mode Register (TMR); bits 191:160 Read Only. 
FEEO 01E0H Trigger Mode Register (TMR); bits 223:192 Read Only. 
FEEO 01FOH Trigger Mode Register (TMR); bits 255:224 Read Only. 
FEEO 0O200H Interrupt Request Register (IRR); bits 31:0 Read Only. 
FEEO 0210H Interrupt Request Register (IRR); bits 63:32 Read Only. 
FEEO 0220H Interrupt Request Register (IRR); bits 95:64 Read Only. 
FEEO 0230H Interrupt Request Register (IRR); bits 127:96 Read Only. 
FEEO 0240H Interrupt Request Register (IRR); bits 159:128 Read Only. 
FEEO 0250H Interrupt Request Register (IRR); bits 191:160 Read Only. 
FEEO 0O260H Interrupt Request Register (IRR); bits 223:192 Read Only. 
FEEO 0270H Interrupt Request Register (IRR); bits 255:224 Read Only. 
FEEO 0280H Error Status Register Read Only. 
FEEO 0290H through Reserved 

FEEO O2E0H 

FEEO O2FOH LWT Corrected Machine Check Interrupt (CMCI) Register Read/Write. 
FEEO 0300H Interrupt Command Register (ICR); bits 0-31 Read/Write. 
FEEO 0310H Interrupt Command Register (ICR); bits 32-63 Read/Write. 
FEEO 0320H LVT Timer Register Read/Write. 
FEEO 0330H LVT Thermal Sensor Register? Read/Write. 
FEEO 0340H LVT Performance Monitoring Counters Register? Read/Write. 
FEEO 0350H LVT LINTO Register Read/Write. 
FEEO 0360H LVT LINT1 Register Read/Write. 
FEEO 0370H LVT Error Register Read/Write. 
FEEO 0380H Initial Count Register (for Timer) Read/Write. 
FEEO 0390H Current Count Register (for Timer) Read Only. 
FEEO O3A0H through Reserved 

FEEO O3D0H 

FEEO O3E0H Divide Configuration Register (for Timer) Read/Write. 
FEEO O3FOH Reserved 

NOTES: 


1. Not supported in the Pentium 4 and Intel Xeon processors. The Illegal Register Access bit (7) of the ESR will not be set when writ- 


ing to these registers. 


2. Introduced in the Pentium 4 and Intel Xeon processors. This APIC register and its associated function are implementation depen- 
dent and may not be present in future IA-32 or Intel 64 processors. 
3. Introduced in the Pentium Pro processor. This APIC register and its associated function are implementation dependent and may not 
be present in future IA-32 or Intel 64 processors. 
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10.4.2 Presence of the Local APIC 


Beginning with the P6 family processors, the presence or absence of an on-chip local APIC can be detected using 
the CPUID instruction. When the CPUID instruction is executed with a source operand of 1 in the EAX register, bit 9 
of the CPUID feature flags returned in the EDX register indicates the presence (set) or absence (clear) of a local 
APIC. 


10.4.3. Enabling or Disabling the Local APIC 
The local APIC can be enabled or disabled in either of two ways: 
1. Using the APIC global enable/disable flag in the IA32_APIC_BASE MSR (MSR address 1BH; see Figure 10-5): 


— When IA32_APIC_BASE[11] is 0, the processor is functionally equivalent to an IA-32 processor without an 
on-chip APIC. The CPUID feature flag for the APIC (see Section 10.4.2, “Presence of the Local APIC”) is also 
set to 0. 


— When IA32_APIC_BASE[11] is set to 0, processor APICs based on the 3-wire APIC bus cannot be generally 
re-enabled until a system hardware reset. The 3-wire bus loses track of arbitration that would be necessary 
for complete re-enabling. Certain APIC functionality can be enabled (for example: performance and 
thermal monitoring interrupt generation). 


— For processors that use Front Side Bus (FSB) delivery of interrupts, software may disable or enable the 
APIC by setting and resetting IA32_APIC_BASE[11]. A hardware reset is not required to re-start APIC 
functionality, if software guarantees no interrupt will be sent to the APIC as IA32_APIC_BASE[11] is 
cleared. 


— When JA32_APIC_BASE[11] is set to 0, prior initialization to the APIC may be lost and the APIC may return 
to the state described in Section 10.4.7.1, “Local APIC State After Power-Up or Reset.” 


2. Using the APIC software enable/disable flag in the spurious-interrupt vector register (see Figure 10-23): 


— If IA32_APIC_BASE[11] is 1, software can temporarily disable a local APIC at any time by clearing the APIC 
software enable/disable flag in the spurious-interrupt vector register (see Figure 10-23). The state of the 
local APIC when in this software-disabled state is described in Section 10.4.7.2, “Local APIC State After It 
Has Been Software Disabled.” 


— When the local APIC is in the software-disabled state, it can be re-enabled at any time by setting the APIC 
software enable/disable flag to 1. 


For the Pentium processor, the APICEN pin (which is shared with the PICD1 pin) is used during power-up or reset 
to disable the local APIC. 


Note that each entry in the LVT has a mask bit that can be used to inhibit interrupts from being delivered to the 
processor from selected local interrupt sources (the LINTO and LINT1 pins, the APIC timer, the performance-moni- 
toring counters, the thermal sensor, and/or the internal APIC error detector). 


10.4.4 Local APIC Status and Location 


The status and location of the local APIC are contained in the IA32_APIC_BASE MSR (see Figure 10-5). MSR bit 

functions are described below: 

° BSP flag, bit 8 — Indicates if the processor is the bootstrap processor (BSP). See Section 8.4, “Multiple- 
Processor (MP) Initialization.” Following a power-up or reset, this flag is set to 1 for the processor selected as 
the BSP and set to 0 for the remaining processors (APs). 

° APIC Global Enable flag, bit 11 — Enables or disables the local APIC (see Section 10.4.3, “Enabling or 
Disabling the Local APIC”). This flag is available in the Pentium 4, Intel Xeon, and P6 family processors. It is not 
guaranteed to be available or available at the same location in future Intel 64 or IA-32 processors. 

° APIC Base field, bits 12 through 35 — Specifies the base address of the APIC registers. This 24-bit value is 
extended by 12 bits at the low end to form the base address. This automatically aligns the address on a 4-KByte 
boundary. Following a power-up or reset, the field is set to FEEO OOOOH. 


° Bits 0 through 7, bits 9 and 10, and bits MAXPHYADDR2 through 63 in the IA32_APIC_BASE MSR are reserved. 
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63 MAXPHYADDR 121110987 0 


Reserved APIC Base | | 


APIC Base—Base physical address 
APIC global enable/disable 
BSP—Processor is BSP 


Reserved 


Figure 10-5. IA32_APIC_BASE MSR (APIC_BASE_MSR in P6 Family) 


10.4.5 Relocating the Local APIC Registers 


The Pentium 4, Intel Xeon, and P6 family processors permit the starting address of the APIC registers to be relo- 
cated from FEEQOOOOH to another physical address by modifying the value in the base address field of the 
IA32_APIC_BASE MSR. This extension of the APIC architecture is provided to help resolve conflicts with memory 
maps of existing systems and to allow individual processors in an MP system to map their APIC registers to 
different locations in physical memory. 


10.4.6 Local APIC ID 


At power up, system hardware assigns a unique APIC ID to each local APIC on the system bus (for Pentium 4 and 
Intel Xeon processors) or on the APIC bus (for P6 family and Pentium processors). The hardware assigned APIC ID 
is based on system topology and includes encoding for socket position and cluster information (see Figure 8-2 and 
Section 8.9.1, “Hierarchical Mapping of Shared Resources”). 


In MP systems, the local APIC ID is also used as a processor ID by the BIOS and the operating system. Some 
processors permit software to modify the APIC ID. However, the ability of software to modify the APIC ID is 
processor model specific. Because of this, operating system software should avoid writing to the local APIC ID 
register. The value returned by bits 31-24 of the EBX register (when the CPUID instruction is executed with a 
source operand value of 1 in the EAX register) is always the Initial APIC ID (determined by the platform initializa- 
tion). This is true even if software has changed the value in the Local APIC ID register. 


The processor receives the hardware assigned APIC ID (or Initial APIC ID) by sampling pins A11# and A12# and 
pins BRO# through BR3# (for the Pentium 4, Intel Xeon, and P6 family processors) and pins BEO# through BE3# 
(for the Pentium processor). The APIC ID latched from these pins is stored in the APIC ID field of the local APIC ID 
register (see Figure 10-6), and is used as the Initial APIC ID for the processor. 


2. The MAXPHYADDR is 36 bits for processors that do not support CPUID leaf 80000008H, or indicated by 
CPUID.8O000008H:EAX|[bits 7:0] for processors that support CPUID leaf 80000008H. 
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P6 family and Pentium processors 
xAPIC Mode 31 27 A 0 
Address: 0FEE0 0020H 
Value after reset: 0000 0000H APIC ID Reserved 
Pentium 4 processors, Xeon processors, and later processors 
31 24 0 
APIC ID Reserved 
x2APIC Mode 31 ) 
MSR Address: 802H x2APIC ID 


Figure 10-6. Local APIC ID Register 


For the P6 family and Pentium processors, the local APIC ID field in the local APIC ID register is 4 bits. Encodings 
OH through EH can be used to uniquely identify 15 different processors connected to the APIC bus. For the Pentium 
4 and Intel Xeon processors, the xAPIC specification extends the local APIC ID field to 8 bits. These can be used to 
identify up to 255 processors in the system. 


10.4.7 Local APIC State 


The following sections describe the state of the local APIC and its registers following a power-up or reset, after the 
local APIC has been software disabled, following an INIT reset, and following an INIT-deassert message. 


x2APIC will introduce 32-bit ID; see Section 10.12. 


10.4.7.1 Local APIC State After Power-Up or Reset 
Following a power-up or reset of the processor, the state of local APIC and its registers are as follows: 
¢ The following registers are reset to all Os. 
¢ IRR, ISR, TMR, ICR, LDR, and TPR. 
¢ Timer initial count and timer current count registers. 
* Divide configuration register. 
¢ The DFR register is reset to all 1s. 
° The LVT register is reset to Os except for the mask bits; these are set to 1s. 
¢ The local APIC version register is not affected. 


¢ The local APIC ID register is set to a unique APIC ID. (Pentium and P6 family processors only). The Arb ID 
register is set to the value in the APIC ID register. 


¢ The spurious-interrupt vector register is initialized to OOOOOOFFH. By setting bit 8 to 0, software disables the 
local APIC. 


e If the processor is the only processor in the system or it is the BSP in an MP system (see Section 8.4.1, “BSP 
and AP Processors”); the local APIC will respond normally to INIT and NMI messages, to INIT# signals and to 
STPCLK# signals. If the processor is in an MP system and has been designated as an AP; the local APIC will 
respond the same as for the BSP. In addition, it will respond to SIPI messages. For P6 family processors only, 
an AP will not respond to a STPCLK# signal. 


10-10 Vol. 3A 


ADVANCED PROGRAMMABLE INTERRUPT CONTROLLER (APIC) 


10.4.7.2 Local APIC State After It Has Been Software Disabled 


When the APIC software enable/disable flag in the spurious interrupt vector register has been explicitly cleared (as 
opposed to being cleared during a power up or reset), the local APIC is temporarily disabled (see Section 10.4.3, 
“Enabling or Disabling the Local APIC”). The operation and response of a local APIC while in this software-disabled 
state is as follows: 


¢ The local APIC will respond normally to INIT, NMI, SMI, and SIPI messages. 
¢ Pending interrupts in the IRR and ISR registers are held and require masking or handling by the CPU. 


¢ The local APIC can still issue IPIs. It is software’s responsibility to avoid issuing IPIs through the IPI mechanism 
and the ICR register if sending interrupts through this mechanism is not desired. 


¢ The reception of any interrupt or transmission of any IPIs that are in progress when the local APIC is disabled 
are completed before the local APIC enters the software-disabled state. 


° The mask bits for all the LVT entries are set. Attempts to reset these bits will be ignored. 


¢ (For Pentium and P6 family processors) The local APIC continues to listen to all bus messages in order to keep 
its arbitration ID synchronized with the rest of the system. 


10.4.7.3_ Local APIC State After an INIT Reset (“Wait-for-SIPI” State) 

An INIT reset of the processor can be initiated in either of two ways: 

¢ By asserting the processor's INIT# pin. 

¢ By sending the processor an INIT IPI (an IPI with the delivery mode set to INIT). 


Upon receiving an INIT through either of these mechanisms, the processor responds by beginning the initialization 
process of the processor core and the local APIC. The state of the local APIC following an INIT reset is the same as 
it is after a power-up or hardware reset, except that the APIC ID and arbitration ID registers are not affected. This 
state is also referred to at the “wait-for-SIPI” state (see also: Section 8.4.2, “MP Initialization Protocol Require- 
ments and Restrictions”). 


10.4.7.4 Local APIC State After It Receives an INIT-Deassert IPI 


Only the Pentium and P6 family processors support the INIT-deassert IPI. An INIT-deassert IPI has no affect on the 
state of the APIC, other than to reload the arbitration ID register with the value in the APIC ID register. 


10.4.8 Local APIC Version Register 


The local APIC contains a hardwired version register. Software can use this register to identify the APIC version 
(see Figure 10-7). In addition, the register specifies the number of entries in the local vector table (LVT) for a 
specific implementation. 


The fields in the local APIC version register are as follows: 


Version The version numbers of the local APIC: 
OXH 82489DxX discrete APIC. 
10H - 15H Integrated APIC. 


Other values reserved. 


Max LVT Entry Shows the number of LVT entries minus 1. For the Pentium 4 and Intel Xeon processors (which 
have 6 LVT entries), the value returned in the Max LVT field is 5; for the P6 family processors 
(which have 5 LVT entries), the value returned is 4; for the Pentium processor (which has 4 LVT 
entries), the value returned is 3. For processors based on the Intel microarchitecture code 
name Nehalem (which has 7 LVT entries) and onward, the value returned is 6. 


Suppress EOI-broadcasts 
Indicates whether software can inhibit the broadcast of EOI message by setting bit 12 of the 
Spurious Interrupt Vector Register; see Section 10.8.5 and Section 10.9. 
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31 25 24 23 16 15 8 7 0 


Reserved Max LVT Entry Reserved Version 


Support for EOl-broadcast suppression 


Value after reset: OBNN OOVVH 

V = Version, N = # of LVT entries minus 1, 

B = 1 if EOl-broadcast suppression supported 
Address: FEEO 0030H 


Figure 10-7. Local APIC Version Register 


10.5 HANDLING LOCAL INTERRUPTS 


The following sections describe facilities that are provided in the local APIC for handling local interrupts. These 
include: the processor’s LINTO and LINT1 pins, the APIC timer, the performance-monitoring counters, the thermal 
sensor, and the internal APIC error detector. Local interrupt handling facilities include: the LVT, the error status 
register (ESR), the divide configuration register (DCR), and the initial count and current count registers. 


10.5.1 Local Vector Table 


The local vector table (LVT) allows software to specify the manner in which the local interrupts are delivered to the 

processor core. It consists of the following 32-bit APIC registers (see Figure 10-8), one for each local interrupt: 

° LVT CMCI Register (FEEO O2FOH) — Specifies interrupt delivery when an overflow condition of corrected 
machine check error count reaching a threshold value occurred in a machine check bank supporting CMCI (see 
Section 15.5.1, “CMCI Local APIC Interface”). 

¢ LVT Timer Register (FEEO 0320H) — Specifies interrupt delivery when the APIC timer signals an interrupt 
(see Section 10.5.4, “APIC Timer”). 


¢ LVT Thermal Monitor Register (FEEO 0330H) — Specifies interrupt delivery when the thermal sensor 
generates an interrupt (see Section 14.8.2, “Thermal Monitor”). This LVT entry is implementation specific, not 
architectural. If implemented, it will always be at base address FEEO 0330H. 

¢ LVT Performance Counter Register (FEEO 0340H) — Specifies interrupt delivery when a performance 
counter generates an interrupt on overflow (see Section 18.6.3.5.8, “Generating an Interrupt on Overflow”). 
This LVT entry is implementation specific, not architectural. If implemented, it is not guaranteed to be at base 
address FEEO 0340H. 

¢ LVTLINTO Register (FEEO 0350H) — Specifies interrupt delivery when an interrupt is signaled at the LINTO 
pin. 

¢ LVT LINT1 Register (FEEO 0360H) — Specifies interrupt delivery when an interrupt is signaled at the LINT1 
pin. 

¢ LVT Error Register (FEEO 0370H) — Specifies interrupt delivery when the APIC detects an internal error 
(see Section 10.5.3, “Error Handling”). 


The LVT performance counter register and its associated interrupt were introduced in the P6 processors and are 
also present in the Pentium 4 and Intel Xeon processors. The LVT thermal monitor register and its associated inter- 
rupt were introduced in the Pentium 4 and Intel Xeon processors. The LVT CMCI register and its associated inter- 
rupt were introduced in the Intel Xeon 5500 processors. 


As shown in Figures 10-8, some of these fields and flags are not available (and reserved) for some entries. 


The setup information that can be specified in the registers of the LVT table is as follows: 
Vector Interrupt vector number. 
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31 19 18 17 16 15 1312 11 8 7 0 
Timer Vector 
A . 
Timer Mode 1 ea ee 
00: One-shot alue after Reset: 
01: Periodic 
10: TSC-Deadline Bellvery statue 
0: Idle 
1: Send Pending 
Maskt 
0: Not Masked 
1: Masked | | 
Interrupt Input Delivery Mode 
Pin Polarity 000: Fixed 
010: SMI 
100: NMI 
Remote 111: ExtINT 
IRR 101: INIT 
All other combinations 
are reserved 
Trigger Mode 
0: Edge 
1: Level 
31 17 voy y 110] 87 0 
CMCI Vector 
LINTO Vector 
LINT1 Vector 
Error Vector 
Performance 
Mon. Counters Vector 
Thermal 
Sensor Vector 
16 15 14 13 12 
R q Address: FEEO 02FOH 
poene Address: FEEO 0350H 
Address: FEEO 0360H 
Address: FEEO 0370H 
t (Pentium 4 and Intel Xeon processors.) When a Address: FEEO 0340H 
performance monitoring counters interrupt is generated, Address: FEEO 0330H 
the mask bit for its associated LVT entry is set. Value After Reset: 0001 0000H 


Figure 10-8. Local Vector Table (LVT) 


Delivery Mode Specifies the type of interrupt to be sent to the processor. Some delivery modes will only 
operate as intended when used in conjunction with a specific trigger mode. The allowable 
delivery modes are as follows: 


000 (Fixed) Delivers the interrupt specified in the vector field. 


010 (SMI) Delivers an SMI interrupt to the processor core through the processor's lo- 
cal SMI signal path. When using this delivery mode, the vector field should 
be set to OOH for future compatibility. 


100 (NMI) Delivers an NMI interrupt to the processor. The vector information is ig- 
nored. 


101 (INIT) Delivers an INIT request to the processor core, which causes the processor 
to perform an INIT. When using this delivery mode, the vector field should 
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be set to OOH for future compatibility. Not supported for the LVT CMCI reg- 
ister, the LVT thermal monitor register, or the LVT performance counter 
register. 


110 Reserved; not supported for any LVT register. 


111 (ExtINT) Causes the processor to respond to the interrupt as if the interrupt origi- 
nated in an externally connected (8259A-compatible) interrupt controller. 
A special INTA bus cycle corresponding to ExtINT, is routed to the external 
controller. The external controller is expected to supply the vector informa- 
tion. The APIC architecture supports only one ExtINT source in a system, 
usually contained in the compatibility bridge. Only one processor in the 
system should have an LVT entry configured to use the ExtINT delivery 
mode. Not supported for the LVT CMCI register, the LVT thermal monitor 
register, or the LVT performance counter register. 


Delivery Status (Read Only) 
Indicates the interrupt delivery status, as follows: 


0 (Idle) There is currently no activity for this interrupt source, or the previous in- 
terrupt from this source was delivered to the processor core and accepted. 


1 (Send Pending) 
Indicates that an interrupt from this source has been delivered to the pro- 
cessor core but has not yet been accepted (see Section 10.5.5, “Local In- 
terrupt Acceptance”). 
Interrupt Input Pin Polarity 
Specifies the polarity of the corresponding interrupt pin: (0) active high or (1) active low. 


Remote IRR Flag (Read Only) 

For fixed mode, level-triggered interrupts; this flag is set when the local APIC accepts the 
interrupt for servicing and is reset when an EOI command is received from the processor. The 
meaning of this flag is undefined for edge-triggered interrupts and other delivery modes. 

Trigger Mode Selects the trigger mode for the local LINTO and LINT1 pins: (0) edge sensitive and (1) level 
sensitive. This flag is only used when the delivery mode is Fixed. When the delivery mode is 
NMI, SMI, or INIT, the trigger mode is always edge sensitive. When the delivery mode is 
ExtINT, the trigger mode is always level sensitive. The timer and error interrupts are always 
treated as edge sensitive. 

If the local APIC is not used in conjunction with an I/O APIC and fixed delivery mode is 
selected; the Pentium 4, Intel Xeon, and P6 family processors will always use level-sensitive 
triggering, regardless if edge-sensitive triggering is selected. 

Software should always set the trigger mode in the LVT LINT1 register to 0 (edge sensitive). 
Level-sensitive interrupts are not supported for LINT1. 

Mask Interrupt mask: (0) enables reception of the interrupt and (1) inhibits reception of the inter- 
rupt. When the local APIC handles a performance-monitoring counters interrupt, it automati- 
cally sets the mask flag in the LVT performance counter register. This flag is set to 1 on reset. 
It can be cleared only by software. 

Timer Mode Bits 18:17 selects the timer mode (see Section 10.5.4): 

(O00b) one-shot mode using a count-down value, 
(01b) periodic mode reloading a count-down value, 


(10b) TSC-Deadline mode using absolute target value in IA32_TSC_DEADLINE MSR (see 
Section 10.5.4.1), 


(11b) is reserved. 


10.5.2 Valid Interrupt Vectors 


The Intel 64 and IA-32 architectures define 256 vector numbers, ranging from 0 through 255 (see Section 6.2, 
“Exception and Interrupt Vectors”). Local and I/O APICs support 240 of these vectors (in the range of 16 to 255) as 
valid interrupts. 
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When an interrupt vector in the range of 0 to 15 is sent or received through the local APIC, the APIC indicates an 
illegal vector in its Error Status Register (see Section 10.5.3, “Error Handling”). The Intel 64 and IA-32 architec- 
tures reserve vectors 16 through 31 for predefined interrupts, exceptions, and Intel-reserved encodings (see Table 
6-1). However, the local APIC does not treat vectors in this range as illegal. 


When an illegal vector value (0 to 15) is written to an LVT entry and the delivery mode is Fixed (bits 8-11 equal 0), 
the APIC may signal an illegal vector error, without regard to whether the mask bit is set or whether an interrupt is 
actually seen on the input. 


10.5.3. Error Handling 


The local APIC records errors detected during interrupt handling in the error status register (ESR). The format of 
the ESR is given in Figure 10-9; it contains the following flags: 


31 876543210 


Reserved 


Illegal Register Address! 


Received Illegal Vector 
Send Illegal Vector 
Redirectable IPI 
Receive Accept Error 
Send Accept Error? 
Receive Checksum Error 
Send Checksum Error? 


Address: FEEO 0280H 
Value after reset: OH 


NOTES: 


1. Used only by Intel Core, Pentium 4, Intel Xeon, and P6 family 
processors; reserved on the Pentium processor. 

2. Used only by some Intel Core and Intel Xeon processors; 
reserved on other processors. 

3. Used only by the P6 family and Pentium processors; 
reserved on Intel Core, Pentium 4 and Intel Xeon processors. 


3 


3 


Figure 10-9. Error Status Register (ESR) 


¢ Bit 0: Send Checksum Error. 
Set when the local APIC detects a checksum error for a message that it sent on the APIC bus. Used only on P6 
family and Pentium processors. 

¢  =Bit 1: Receive Checksum Error. 
Set when the local APIC detects a checksum error for a message that it received on the APIC bus. Used only on 
P6 family and Pentium processors. 

¢ Bit 2: Send Accept Error. 
Set when the local APIC detects that a message it sent was not accepted by any APIC on the APIC bus. Used 
only on P6 family and Pentium processors. 

¢ ~=Bit 3: Receive Accept Error. 
Set when the local APIC detects that the message it received was not accepted by any APIC on the APIC bus, 
including itself. Used only on P6 family and Pentium processors. 

° ~=Bit 4: Redirectable IPI. 
Set when the local APIC detects an attempt to send an IPI with the lowest-priority delivery mode and the local 
APIC does not support the sending of such IPIs. This bit is used on some Intel Core and Intel Xeon processors. 
As noted in Section 10.6.2, the ability of a processor to send a lowest-priority IPI is model-specific and should 
be avoided. 
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° Bit 5: Send Illegal Vector. 
Set when the local APIC detects an illegal vector (one in the range 0 to 15) in the message that it is sending. 
This occurs as the result of a write to the ICR (in both xAPIC and x2APIC modes) or to SELF IPI register (x2APIC 
mode only) with an illegal vector. 


If the local APIC does not support the sending of lowest-priority IPIs and software writes the ICR to send a 
lowest-priority IPI with an illegal vector, the local APIC sets only the “redirectable IPI” error bit. The interrupt is 
not processed and hence the “Send Illegal Vector” bit is not set in the ESR. 


° Bit 6: Receive Illegal Vector. 
Set when the local APIC detects an illegal vector (one in the range 0 to 15) in an interrupt message it receives 
or in an interrupt generated locally from the local vector table or via a self IPI. Such interrupts are not delivered 
to the processor; the local APIC will never set an IRR bit in the range 0 to 15. 


° Bit 7: Illegal Register Address 
Set when the local APIC is in xAPIC mode and software attempts to access a register that is reserved in the 
processor's local-APIC register-address space; see Table 10-1. (The local-APIC register-address space 
comprises the 4 KBytes at the physical address specified in the IA32_APIC_BASE MSR.) Used only on Intel 
Core, Intel Atom™, Pentium 4, Intel Xeon, and P6 family processors. 


In x2APIC mode, software accesses the APIC registers using the RDMSR and WRMSR instructions. Use of one 
of these instructions to access a reserved register cause a general-protection exception (see Section 
10.12.1.3). They do not set the “Illegal Register Access” bit in the ESR. 


The ESR is a write/read register. Before attempt to read from the ESR, software should first write to it. (The value 
written does not affect the values read subsequently; only zero may be written in x2APIC mode.) This write clears 
any previously logged errors and updates the ESR with any errors detected since the last write to the ESR. This 
write also rearms the APIC error interrupt triggering mechanism. 


The LVT Error Register (see Section 10.5.1) allows specification of the vector of the interrupt to be delivered to the 
processor core when APIC error is detected. The register also provides a means of masking an APIC-error interrupt. 
This masking only prevents delivery of APIC-error interrupts; the APIC continues to record errors in the ESR. 


10.5.4 APIC Timer 


The local APIC unit contains a 32-bit programmable timer that is available to software to time events or operations. 
This timer is set up by programming four registers: the divide configuration register (see Figure 10-10), the initial- 
count and current-count registers (see Figure 10-11), and the LVT timer register (see Figure 10-8). 


If CPUID.O6H:EAX.ARAT[bit 2] = 1, the processor’s APIC timer runs at a constant rate regardless of P-state transi- 
tions and it continues to run at the same rate in deep C-states. 


If CPUID.O6H:EAX.ARAT[bit 2] = 0 or if CPUID 06H is not supported, the APIC timer may temporarily stop while the 
processor is in deep C-states or during transitions caused by Enhanced Intel SpeedStep® Technology. 


31 43210 


Reserved 0 


Address: FEEO 03EOH 
Value after reset: OH 


Divide Value (bits 0, 1 and 3) ___| 
000: Divide by 2 

001: Divide by 4 

010: Divide by 8 

011: Divide by 16 

100: Divide by 32 

101: Divide by 64 

110: Divide by 128 

111: Divide by 1 


Figure 10-10. Divide Configuration Register 


The APIC timer frequency will be the processor's bus clock or core crystal clock frequency (when TSC/core crystal 
clock ratio is enumerated in CPUID leaf 0x15) divided by the value specified in the divide configuration register. 


10-16 Vol. 3A 


ADVANCED PROGRAMMABLE INTERRUPT CONTROLLER (APIC) 


31 0 


Initial Count 


Current Count 


Address: Initial Count FEEO 0380H 
Current Count FEEO 0390H 
Value after reset: OH 


Figure 10-11. Initial Count and Current Count Registers 


The timer can be configured through the timer LVT entry for one-shot or periodic operation. In one-shot mode, the 
timer is started by programming its initial-count register. The initial count value is then copied into the current- 
count register and count-down begins. After the timer reaches zero, a timer interrupt is generated and the timer 
remains at its 0 value until reprogrammed. 


In periodic mode, the timer is started by writing to the initial-count register (as in one-shot mode), and the value 
written is copied into the current-count register, which counts down. The current-count register is automatically 
reloaded from the initial-count register when the count reaches 0 and a timer interrupt is generated, and the 
count-down is repeated. If during the count-down process the initial-count register is set, counting will restart, 
using the new initial-count value. The initial-count register is a read-write register; the current-count register is 
read only. 


A write of 0 to the initial-count register effectively stops the local APIC timer, in both one-shot and periodic mode. 


The LVT timer register determines the vector number that is delivered to the processor with the timer interrupt that 
is generated when the timer count reaches zero. The mask flag in the LVT timer register can be used to mask the 
timer interrupt. 


NOTE 


Changing the mode of the APIC timer (from one-shot to periodic or vice versa) by writing to the 
timer LVT entry does not start the timer. To start the timer, it is necessary to write to the initial- 
count register as described above. 


10.5.4.1 TSC-Deadline Mode 
The mode of operation of the local-APIC timer is determined by the LVT Timer Register. Specifically: 
° If CPUID.01H:ECX.TSC_Deadline[bit 24] = 0, the mode is determined by bit 17 of the register. 


¢ If CPUID.01H:ECX.TSC_Deadline[bit 24] = 1, the mode is determined by bits 18:17. See Figure 10-8. (If 
CPUID.01H:ECX.TSC_Deadline[bit 24] = 0, bit 18 of the register is reserved.) 


The supported timer modes are given in Table 10-2. The three modes of the local APIC timer are mutually exclu- 
sive. 


Table 10-2. Local APIC Timer Modes 


LWT Bits [18:17] Timer Mode 

O00b One-shot mode, program count-down value in an initial-count register. See Section 10.5.4 
01b Periodic mode, program interval value in an initial-count register. See Section 10.5.4 

10b TSC-Deadline mode, program target value in IA32_TSC_DEADLINE MSR. 

11b Reserved 


TSC-deadline mode allows software to use the local APIC timer to signal an interrupt at an absolute time. In TSC- 
deadline mode, writes to the initial-count register are ignored; and current-count register always reads 0. Instead, 
timer behavior is controlled using the IA32_TSC_DEADLINE MSR. 
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The IA32_TSC_DEADLINE MSR (MSR address 6E0H) is a per-logical processor MSR that specifies the time at which 
a timer interrupt should occur. Writing a non-zero 64-bit value into IA32_TSC_DEADLINE arms the timer. An inter- 
rupt is generated when the logical processor’s time-stamp counter equals or exceeds the target value in the 
IA32_TSC_DEADLINE MSR.? When the timer generates an interrupt, it disarms itself and clears the 
IA32_TSC_DEADLINE MSR. Thus, each write to the IA32_TSC_DEADLINE MSR generates at most one timer inter- 
rupt. 


In TSC-deadline mode, writing 0 to the IA32_TSC_DEADLINE MSR disarms the local-APIC timer. Transitioning 
between TSC-deadline mode and other timer modes also disarms the timer. 


The hardware reset value of the IA32_TSC_DEADLINE MSR is 0. In other timer modes (LVT bit 18 = 0), the 
IA32_TSC_DEADLINE MSR reads zero and writes are ignored. 


Software can configure the TSC-deadline timer to deliver a single interrupt using the following algorithm: 
1. Detect support for TSC-deadline mode by verifying CPUID.1:ECX.24 = 1. 
2. Select the TSC-deadline mode by programming bits 18:17 of the LVT Timer register with 10b. 


3. Program the IA32_TSC_DEADLINE MSR with the target TSC value at which the timer interrupt is desired. This 
causes the processor to arm the timer. 


4. The processor generates a timer interrupt when the value of time-stamp counter is greater than or equal to that 
of IA32_TSC_DEADLINE. It then disarms the timer and clear the IA32_TSC_DEADLINE MSR. (Both the time- 
stamp counter and the IA32_TSC_DEADLINE MSR are 64-bit unsigned integers.) 


5. Software can re-arm the timer by repeating step 3. 
The following are usage guidelines for TSC-deadline mode: 


¢ Writes to the IA32_TSC_DEADLINE MSR are not serialized. Therefore, system software should not use WRMSR 
to the IA32_TSC_DEADLINE MSR as a serializing instruction. Read and write accesses to the 
IA32_TSC_DEADLINE and other MSR registers will occur in program order. 


¢* Software can disarm the timer at any time by writing 0 to the IA32_TSC_DEADLINE MSR. 


e If timer is armed, software can change the deadline (forward or backward) by writing a new value to the 
IA32_TSC_DEADLINE MSR. 


¢ If software disarms the timer or postpones the deadline, race conditions may result in the delivery of a spurious 
timer interrupt. Software is expected to detect such spurious interrupts by checking the current value of the 
time-stamp counter to confirm that the interrupt was desired.? 


¢ In xAPIC mode (in which the local-APIC registers are memory-mapped), software must order the memory- 
mapped write to the LVT entry that enables TSC-deadline mode and any subsequent WRMSR to the 
IA32_TSC_DEADLINE MSR. Software can assure proper ordering by executing the MFENCE instruction after the 
memory-mapped write and before any WRMSR. (In x2APIC mode, the WRMSR instruction is used to write to 
the LVT entry. The processor ensures the ordering of this write and any subsequent WRMSR to the deadline; no 
fencing is required.) 


10.5.5 —_ Local Interrupt Acceptance 


When a local interrupt is sent to the processor core, it is subject to the acceptance criteria specified in the interrupt 
acceptance flow chart in Figure 10-17. If the interrupt is accepted, it is logged into the IRR register and handled by 
the processor according to its priority (see Section 10.8.4, “Interrupt Acceptance for Fixed Interrupts”). If the 
interrupt is not accepted, it is sent back to the local APIC and retried. 


3. If the logical processor is in VMX non-root operation, a read of the time-stamp counter (using either RDMSR, RDTSC, or RDTSCP) may 
not return the actual value of the time-stamp counter; see Chapter 24 of the Intel® 64 and IA-32 Architectures Software Devel- 
oper’s Manual, Volume 3C. It is the responsibility of software operating in VMX root operation to coordinate the virtualization of the 
time-stamp counter and the [A32_TSC_DEADLINE MSR. 
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10.6 ISSUING INTERPROCESSOR INTERRUPTS 


The following sections describe the local APIC facilities that are provided for issuing interprocessor interrupts (IPIs) 
from software. The primary local APIC facility for issuing IPIs is the interrupt command register (ICR). The ICR can 


be used for the following functions: 
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* To send an interrupt to another processor. 

¢* To allow a processor to forward an interrupt that it received but did not service to another processor for 
servicing. 

¢® To direct the processor to interrupt itself (perform a self interrupt). 

° To deliver special IPIs, such as the start-up IPI (SIPI) message, to other processors. 


Interrupts generated with this facility are delivered to the other processors in the system through the system bus 
(for Pentium 4 and Intel Xeon processors) or the APIC bus (for P6 family and Pentium processors). The ability for a 
processor to send a lowest priority IPI is model specific and should be avoided by BIOS and operating system soft- 
ware. 


10.6.1. Interrupt Command Register (ICR) 


The interrupt command register (ICR) is a 64-bit* local APIC register (see Figure 10-12) that allows software 
running on the processor to specify and send interprocessor interrupts (IPIs) to other processors in the system. 


To send an IPI, software must set up the ICR to indicate the type of IPI message to be sent and the destination 
processor or processors. (All fields of the ICR are read-write by software with the exception of the delivery status 
field, which is read-only.) The act of writing to the low doubleword of the ICR causes the IPI to be sent. 


63 56 55 32 


Destination Field Reserved 


31 2019181716 1514 1312 1110 87 0 


Vector 


Reserved 


Destination Shorthand Delivery Mode 
00: No Shorthand 000: Fixed 
01: Self 001: Lowest Priority’ 
10: All Including Self 010: SMI 
11: All Excluding Self 011: Reserved 
100: NMI 
101: INIT 
110: Start Up 
Reserved 111: Reserved 


Destination Mode 
0: Physical 
1: Logical 


Delivery Status 
0: Idle 
1: Send Pending 


Level 
Address: FEEO 0300H (0 - 31) 0 = De-assert 
FEEO 0310H (32 - 63) 1 = Assert 


Value after Reset: OH 


Trigger Mode 
0: Edge 
1: Level 


NOTE: 
1. The ability of a processor to send Lowest Priority IPI is model specific. 


Figure 10-12. Interrupt Command Register (ICR) 


4. In XAPIC mode the ICR is addressed as two 32-bit registers, ICR_LOW (FFEO 0300H) and ICR_HIGH (FFEO 0310H). In x2APIC mode, 
the ICR uses MSR 830H. 
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The vector number of the interrupt being sent. 
Specifies the type of IPI to be sent. This field is also know as the IPI message type field. 


000 (Fixed) 


Delivers the interrupt specified in the vector field to the target processor or 
processors. 


001 (Lowest Priority) 


010 (SMI) 


Same as fixed mode, except that the interrupt is delivered to the proces- 
sor executing at the lowest priority among the set of processors specified 
in the destination field. The ability for a processor to send a lowest priority 
IPI is model specific and should be avoided by BIOS and operating system 
software. 


Delivers an SMI interrupt to the target processor or processors. The vector 
field must be programmed to OOH for future compatibility. 


011 (Reserved) 


100 (NMI) 


101 (INIT) 


Delivers an NMI interrupt to the target processor or processors. The vector 
information is ignored. 


Delivers an INIT request to the target processor or processors, which 
causes them to perform an INIT. As a result of this IPI message, all the tar- 
get processors perform an INIT. The vector field must be programmed to 
OOH for future compatibility. 


101 (INIT Level De-assert) 


(Not supported in the Pentium 4 and Intel Xeon processors.) Sends a syn- 
chronization message to all the local APICs in the system to set their arbi- 
tration IDs (stored in their Arb ID registers) to the values of their APIC IDs 
(see Section 10.7, “System and APIC Bus Arbitration”). For this delivery 
mode, the level flag must be set to 0 and trigger mode flag to 1. This IPI is 
sent to all processors, regardless of the value in the destination field or the 
destination shorthand field; however, software should specify the “all in- 
cluding self” shorthand. 


110 (Start-Up) 


Sends a special “start-up” IPI (called a SIPI) to the target processor or 
processors. The vector typically points to a start-up routine that is part of 
the BIOS boot-strap code (see Section 8.4, “Multiple-Processor (MP) Ini- 
tialization”). IPIs sent with this delivery mode are not automatically retried 
if the source APIC is unable to deliver it. It is up to the software to deter- 
mine if the SIPI was not successfully delivered and to reissue the SIPI if 
necessary. 


Destination Mode Selects either physical (0) or logical (1) destination mode (see Section 10.6.2, “Determining 


IPI Destination”). 


Delivery Status (Read Only) 
Indicates the IPI delivery status, as follows: 


Level 


0 (Idle) 


Indicates that this local APIC has completed sending any previous IPIs. 


1 (Send Pending) 


Indicates that this local APIC has not completed sending the last IPI. 


For the INIT level de-assert delivery mode this flag must be set to 0; for all other delivery 
modes it must be set to 1. (This flag has no meaning in Pentium 4 and Intel Xeon processors, 
and will always be issued as a 1.) 
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Trigger Mode 


Selects the trigger mode when using the INIT level de-assert delivery mode: edge (0) or level 
(1). It is ignored for all other delivery modes. (This flag has no meaning in Pentium 4 and Intel 
Xeon processors, and will always be issued as a 0.) 


Destination Shorthand 


Destination 


Indicates whether a shorthand notation is used to specify the destination of the interrupt and, 
if so, which shorthand is used. Destination shorthands are used in place of the 8-bit destina- 
tion field, and can be sent by software using a single write to the low doubleword of the ICR. 
Shorthands are defined for the following cases: software self interrupt, IPIs to all processors in 
the system including the sender, IPIs to all processors in the system excluding the sender. 


00: (No Shorthand) 
The destination is specified in the destination field. 


01: (Self) The issuing APIC is the one and only destination of the IPI. This destination 
shorthand allows software to interrupt the processor on which it is execut- 
ing. An APIC implementation is free to deliver the self-interrupt message 
internally or to issue the message to the bus and “snoop” it as with any 
other IPI message. 


10: (All Including Self) 
The IPI is sent to all processors in the system including the processor send- 
ing the IPI. The APIC will broadcast an IPI message with the destination 
field set to FH for Pentium and P6 family processors and to FFH for Pentium 
4 and Intel Xeon processors. 


11: (All Excluding Self) 

The IPI is sent to all processors in a system with the exception of the pro- 
cessor sending the IPI. The APIC broadcasts a message with the physical 
destination mode and destination field set to FH for Pentium and P6 family 
processors and to FFH for Pentium 4 and Intel Xeon processors. Support 
for this destination shorthand in conjunction with the lowest-priority deliv- 
ery mode is model specific. For Pentium 4 and Intel Xeon processors, when 
this shorthand is used together with lowest priority delivery mode, the IPI 
may be redirected back to the issuing processor. 


Specifies the target processor or processors. This field is only used when the destination short- 
hand field is set to OOB. If the destination mode is set to physical, then bits 56 through 59 
contain the APIC ID of the target processor for Pentium and P6 family processors and bits 56 
through 63 contain the APIC ID of the target processor the for Pentium 4 and Intel Xeon 
processors. If the destination mode is set to logical, the interpretation of the 8-bit destination 
field depends on the settings of the DFR and LDR registers of the local APICs in all the proces- 
sors in the system (see Section 10.6.2, “Determining IPI Destination”). 


Not all combinations of options for the ICR are valid. Table 10-3 shows the valid combinations for the fields in the 
ICR for the Pentium 4 and Intel Xeon processors; Table 10-4 shows the valid combinations for the fields in the ICR 
for the P6 family processors. Also note that the lower half of the ICR may not be preserved over transitions to the 


deepest C-States. 


ICR operation in x2APIC mode is discussed in Section 10.12.9. 
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Table 10-3 Valid Combinations for the Pentium 4 and Intel Xeon Processors’ 
Local xAPIC Interrupt Command Register 


Destination Shorthand | Valid/Invalid | Trigger Mode | Delivery Mode Destination Mode 
No Shorthand Valid Edge All Modes! Physical or Logical 
No Shorthand Invalid@ Level All Modes Physical or Logical 
Self Valid Edge Fixed x3 

Self Invalid? Level Fixed : 

Self Invalid X Lowest Priority, NMI, INIT, SMI, Start-Up X 

All Including Self Valid Edge Fixed X 

All Including Self Invalid@ Level Fixed X 

All Including Self Invalid X Lowest Priority, NMI, INIT, SMI, Start-Up X 

All Excluding Self Valid Edge Fixed, Lowest Priority !“4, NMI, INIT, SMI, Start-Up X 

All Excluding Self Invalid? Level Fixed, Lowest Priority*, NMI, INIT, SMI, Start-Up X 


NOTES: 


1. The ability of a processor to send a lowest priority IPI is model specific. 


2. For these interrupts, if the trigger mode bit is 1 (Level), the local xAPIC will override the bit setting and issue the interrupt as an 
edge triggered interrupt. 


3. X means the setting is ignored. 


4. When using the “lowest priority” delivery mode and the “all excluding self” destination, the IPI can be redirected back to the issuing 
APIC, which is essentially the same as the “all including self” destination mode. 


Table 10-4 Valid Combinations for the P6 Family Processors’ Local APIC Interrupt Command Register 


Destination Shorthand | Valid/Invalid | Trigger Mode | Delivery Mode Destination Mode 
No Shorthand Valid Edge All Modes! Physical or Logical 
No Shorthand Valid@ Level Fixed, Lowest Priority!, NMI Physical or Logical 
No Shorthand Valid? Level INIT Physical or Logical 
Self Valid Edge Fixed x 

Self Valid@ Level Fixed ~ 

Self Invalid? X Lowest Priority, NMI, INIT, SMI, Start-Up X 

All including Self Valid Edge Fixed X 

All including Self Valid@ Level Fixed X 

All including Self Invalid? X Lowest Priority, NMI, INIT, SMI, Start-Up X 

All excluding Self Valid Edge All Modes! X 

All excluding Self Valid@ Level Fixed, Lowest Priority!, NMI X 

All excluding Self Invalid? Level SMI, Start-Up X 

All excluding Self Valid? Level INIT X 

X Invalid? Level SMI, Start-Up X 

NOTES: 


1. The ability of a processor to send a lowest priority IPI is model specific. 
2. Treated as edge triggered if level bit is set to 1, otherwise ignored. 


3. Treated as edge triggered when Level bit is set to 1; treated as “INIT Level Deassert” message when level bit is set to O (deassert). 
Only INIT level deassert messages are allowed to have the level bit set to O. For all other messages the level bit must be set to 1. 


4. X means the setting is ignored. 


5. The behavior of the APIC is undefined. 
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10.6.2 Determining IPI Destination 


The destination of an IPI> can be one, all, or a subset (group) of the processors on the system bus. The sender of 
the IPI specifies the destination of an IPI with the following APIC registers and fields within the registers: 


¢ ICR Register — The following fields in the ICR register are used to specify the destination of an IPI. 
— Destination Mode — Selects one of two destination modes (physical or logical). 


— Destination Field — In physical destination mode, used to specify the APIC ID of the destination 
processor; in logical destination mode, used to specify a message destination address (MDA) that can be 
used to select specific processors in clusters. 


— Destination Shorthand — A quick method of specifying all processors, all excluding self, or self as the 
destination. 


— Delivery mode, Lowest Priority — Architecturally specifies that a lowest-priority arbitration mechanism 
be used to select a destination processor from a specified group of processors. The ability of a processor to 
send a lowest priority IPI is model specific and should be avoided by BIOS and operating system software. 


* Local destination register (LDR) — Used in conjunction with the logical destination mode and MDAs to 
select the destination processors. 


* Destination format register (DFR) — Used in conjunction with the logical destination mode and MDAs to 
select the destination processors. 


How the ICR, LDR, and DFR are used to select an IPI destination depends on the destination mode used: physical, 
logical, broadcast/self, or lowest-priority delivery mode. These destination modes are described in the following 
sections. 


10.6.2.1 Physical Destination Mode 


In physical destination mode, the destination processor is specified by its local APIC ID (see Section 10.4.6, “Local 
APIC ID”). For Pentium 4 and Intel Xeon processors, either a single destination (local APIC IDs OOH through FEH) 
or a broadcast to all APICs (the APIC ID is FFH) may be specified in physical destination mode. 


A broadcast IPI (bits 28-31 of the MDA are 1's) or I/O subsystem initiated interrupt with lowest priority delivery 
mode is not supported in physical destination mode and must not be configured by software. Also, for any non- 
broadcast IPI or I/O subsystem initiated interrupt with lowest priority delivery mode, software must ensure that 
APICs defined in the interrupt address are present and enabled to receive interrupts. 


For the P6 family and Pentium processors, a single destination is specified in physical destination mode with a local 
APIC ID of OH through OEH, allowing up to 15 local APICs to be addressed on the APIC bus. A broadcast to all local 
APICs is specified with OFH. 


NOTE 


The number of local APICs that can be addressed on the system bus may be restricted by hardware. 


10.6.2.2 Logical Destination Mode 


In logical destination mode, IPI destination is specified using an 8-bit message destination address (MDA), which is 
entered in the destination field of the ICR. Upon receiving an IPI message that was sent using logical destination 
mode, a local APIC compares the MDA in the message with the values in its LDR and DFR to determine if it should 
accept and handle the IPI. For both configurations of logical destination mode, when combined with lowest priority 
delivery mode, software is responsible for ensuring that all of the local APICs included in or addressed by the IPI or 
I/O subsystem interrupt are present and enabled to receive the interrupt. 


Figure 10-13 shows the layout of the logical destination register (LDR). The 8-bit logical APIC ID field in this 
register is used to create an identifier that can be compared with the MDA. 


5. Determination of IPI destinations in x2APIC mode is discussed in Section 10.12.10. 
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NOTE 


The logical APIC ID should not be confused with the local APIC ID that is contained in the local APIC 
ID register. 


31 24 23 0 


Logical APIC ID Reserved 


Address: OFEEO 0ODOH 
Value after reset: 0000 OOOOH 


Figure 10-13. Logical Destination Register (LDR) 


Figure 10-14 shows the layout of the destination format register (DFR). The 4-bit model field in this register selects 
one of two models (flat or cluster) that can be used to interpret the MDA when using logical destination mode. 


31 28 0 


Model Reserved (All 1s) 


Flat model: 1111B 
Cluster model: 0000B 


Address: OFEEO 00EOH 
Value after reset: FFFF FFFFH 


Figure 10-14. Destination Format Register (DFR) 


The interpretation of MDA for the two models is described in the following paragraphs. 


1. 


Flat Model — This model is selected by programming DFR bits 28 through 31 to 1111. Here, a unique logical 
APIC ID can be established for up to 8 local APICs by setting a different bit in the logical APIC ID field of the LDR 
for each local APIC. A group of local APICs can then be selected by setting one or more bits in the MDA. 


Each local APIC performs a bit-wise AND of the MDA and its logical APIC ID. If a true condition (non-zero) is 
detected, the local APIC accepts the IPI message. A broadcast to all APICs is achieved by setting the MDA to 1s. 


Cluster Model — This model is selected by programming DFR bits 28 through 31 to 0000. This model supports 
two basic destination schemes: flat cluster and hierarchical cluster. 


The flat cluster destination model is only supported for P6 family and Pentium processors. Using this model, all 
APICs are assumed to be connected through the APIC bus. Bits 60 through 63 of the MDA contains the encoded 
address of the destination cluster and bits 56 through 59 identify up to four local APICs within the cluster (each 
bit is assigned to one local APIC in the cluster, as in the flat connection model). To identify one or more local 
APICs, bits 60 through 63 of the MDA are compared with bits 28 through 31 of the LDR to determine if a local 
APIC is part of the cluster. Bits 56 through 59 of the MDA are compared with Bits 24 through 27 of the LDR to 
identify a local APICs within the cluster. 


Sets of processors within a cluster can be specified by writing the target cluster address in bits 60 through 63 
of the MDA and setting selected bits in bits 56 through 59 of the MDA, corresponding to the chosen members 
of the cluster. In this mode, 15 clusters (with cluster addresses of 0 through 14) each having 4 local APICs can 
be specified in the message. For the P6 and Pentium processor’s local APICs, however, the APIC arbitration 
ID supports only 15 APIC agents. Therefore, the total number of processors and their local APICs supported 
in this mode is limited to 15. Broadcast to all local APICs is achieved by setting all destination bits to one. This 
guarantees a match on all clusters and selects all APICs in each cluster. A broadcast IPI or I/O subsystem 
broadcast interrupt with lowest priority delivery mode is not supported in cluster mode and must not be 
configured by software. 


The hierarchical cluster destination model can be used with Pentium 4, Intel Xeon, P6 family, or Pentium 
processors. With this model, a hierarchical network can be created by connecting different flat clusters via 
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independent system or APIC buses. This scheme requires a cluster manager within each cluster, which is 
responsible for handling message passing between system or APIC buses. One cluster contains up to 4 agents. 
Thus 15 cluster managers, each with 4 agents, can form a network of up to 60 APIC agents. Note that hierar- 
chical APIC networks requires a special cluster manager device, which is not part of the local or the I/O APIC 
units. 


NOTES 


All processors that have their APIC software enabled (using the spurious vector enable/disable bit) 
must have their DFRs (Destination Format Registers) programmed identically. 

The default mode for DFR is flat mode. If you are using cluster mode, DFRs must be programmed 
before the APIC is software enabled. Since some chipsets do not accurately track a system view of 
the logical mode, program DFRs as soon as possible after starting the processor. 


10.6.2.3. Broadcast/Self Delivery Mode 


The destination shorthand field of the ICR allows the delivery mode to be by-passed in favor of broadcasting the IPI 
to all the processors on the system bus and/or back to itself (see Section 10.6.1, “Interrupt Command Register 
(ICR)”). Three destination shorthands are supported: self, all excluding self, and all including self. The destination 
mode is ignored when a destination shorthand is used. 


10.6.2.4 Lowest Priority Delivery Mode 


With lowest priority delivery mode, the ICR is programmed to send an IPI to several processors on the system bus, 
using the logical or shorthand destination mechanism for selecting the processor. The selected processors then 
arbitrate with one another over the system bus or the APIC bus, with the lowest-priority processor accepting the 
IPI. 


For systems based on the Intel Xeon processor, the chipset bus controller accepts messages from the I/O APIC 
agents in the system and directs interrupts to the processors on the system bus. When using the lowest priority 
delivery mode, the chipset chooses a target processor to receive the interrupt out of the set of possible targets. The 
Pentium 4 processor provides a special bus cycle on the system bus that informs the chipset of the current task 
priority for each logical processor in the system. The chipset saves this information and uses it to choose the lowest 
priority processor when an interrupt is received. 


For systems based on P6 family processors, the processor priority used in lowest-priority arbitration is contained in 
the arbitration priority register (APR) in each local APIC. Figure 10-15 shows the layout of the APR. 


31 87 43 0 


Reserved 


Arbitration Priority Class 
Address: FEEO 0090H Arbitration Priority Sub-Class 
Value after reset: OH 


Figure 10-15. Arbitration Priority Register (APR) 


The APR value is computed as follows: 


IF (TPR[7:4] = IRRV[7:4]) AND (TPR[7:4] > ISRV[7:4]) 
THEN 
APR[7:0] <— TPR[7:0] 
ELSE 
APR[7:4] — max(TPR[7:4] AND ISRV[7:4], IRRV[7:4]) 
APR[3:0] < 0. 
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Here, the TPR value is the task priority value in the TPR (see Figure 10-18), the IRRV value is the vector number 
for the highest priority bit that is set in the IRR (see Figure 10-20) or OOH (if no IRR bit is set), and the ISRV value 
is the vector number for the highest priority bit that is set in the ISR (see Figure 10-20). Following arbitration 
among the destination processors, the processor with the lowest value in its APR handles the IPI and the other 
processors ignore it. 


(P6 family and Pentium processors.) For these processors, if a focus processor exists, it may accept the interrupt, 
regardless of its priority. A processor is said to be the focus of an interrupt if it is currently servicing that interrupt 
or if it has a pending request for that interrupt. For Intel Xeon processors, the concept of a focus processor is not 
supported. 


In operating systems that use the lowest priority delivery mode but do not update the TPR, the TPR information 
saved in the chipset will potentially cause the interrupt to be always delivered to the same processor from the 
logical set. This behavior is functionally backward compatible with the P6 family processor but may result in unex- 
pected performance implications. 


10.6.3 ‘IPI Delivery and Acceptance 


When the low double-word of the ICR is written to, the local APIC creates an IPI message from the information 
contained in the ICR and sends the message out on the system bus (Pentium 4 and Intel Xeon processors) or the 
APIC bus (P6 family and Pentium processors). The manner in which these IPIs are handled after being issues in 
described in Section 10.8, “Handling Interrupts.” 


10.7. SYSTEM AND APIC BUS ARBITRATION 


When several local APICs and the I/O APIC are sending IPI and interrupt messages on the system bus (or APIC 
bus), the order in which the messages are sent and handled is determined through bus arbitration. 


For the Pentium 4 and Intel Xeon processors, the local and I/O APICs use the arbitration mechanism defined for the 
system bus to determine the order in which IPIs are handled. This mechanism is non-architectural and cannot be 
controlled by software. 


For the P6 family and Pentium processors, the local and I/O APICs use an APIC-based arbitration mechanism to 
determine the order in which IPIs are handled. Here, each local APIC is given an arbitration priority of from 0 to 15, 
which the I/O APIC uses during arbitration to determine which local APIC should be given access to the APIC bus. 
The local APIC with the highest arbitration priority always wins bus access. Upon completion of an arbitration 
round, the winning local APIC lowers its arbitration priority to 0 and the losing local APICs each raise theirs by 1. 


The current arbitration priority for a local APIC is stored in a 4-bit, software-transparent arbitration ID (Arb ID) 
register. During reset, this register is initialized to the APIC ID number (stored in the local APIC ID register). The 
INIT level-deassert IPI, which is issued with and ICR command, can be used to resynchronize the arbitration prior- 
ities of the local APICs by resetting Arb ID register of each agent to its current APIC ID value. (The Pentium 4 and 
Intel Xeon processors do not implement the Arb ID register. ) 


Section 10.10, “APIC Bus Message Passing Mechanism and Protocol (P6 Family, Pentium Processors),” describes 
the APIC bus arbitration protocols and bus message formats, while Section 10.6.1, “Interrupt Command Register 
(ICR),” describes the INIT level de-assert IPI message. 


Note that except for the SIPI IPI (see Section 10.6.1, “Interrupt Command Register (ICR)”), all bus messages that 
fail to be delivered to their specified destination or destinations are automatically retried. Software should avoid 
situations in which IPIs are sent to disabled or nonexistent local APICs, causing the messages to be resent repeat- 
edly. Additionally, interrupt sources that target the APIC should be masked or changed to no longer target the 
APIC. 


10.8 HANDLING INTERRUPTS 


When a local APIC receives an interrupt from a local source, an interrupt message from an I/O APIC, or an IPI, the 
manner in which it handles the message depends on processor implementation, as described in the following 
sections. 
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10.8.1. = Interrupt Handling with the Pentium 4 and Intel Xeon Processors 


With the Pentium 4 and Intel Xeon processors, the local APIC handles the local interrupts, interrupt messages, and 
IPIs it receives as follows: 


1. It determines if it is the specified destination or not (see Figure 10-16). If it is the specified destination, it 
accepts the message; if it is not, it discards the message. 


Wait to Receive 
Bus Message 


, Belong to 
Discard No Destination? 


Message 


Accept 
Message 


Figure 10-16. Interrupt Acceptance Flow Chart for the Local APIC (Pentium 4 and Intel Xeon Processors) 


2. If the local APIC determines that it is the designated destination for the interrupt and if the interrupt request is 
an NMI, SMI, INIT, ExtINT, or SIPI, the interrupt is sent directly to the processor core for handling. 


3. If the local APIC determines that it is the designated destination for the interrupt but the interrupt request is 
not one of the interrupts given in step 2, the local APIC sets the appropriate bit in the IRR. 


4. When interrupts are pending in the IRR register, the local APIC dispatches them to the processor one at a time, 
based on their priority and the current processor priority in the PPR (see Section 10.8.3.1, “Task and Processor 
Priorities”). 

5. When a fixed interrupt has been dispatched to the processor core for handling, the completion of the handler 
routine is indicated with an instruction in the instruction handler code that writes to the end-of-interrupt (EOI) 
register in the local APIC (see Section 10.8.5, “Signaling Interrupt Servicing Completion”). The act of writing to 
the EOI register causes the local APIC to delete the interrupt from its ISR queue and (for level-triggered 
interrupts) send a message on the bus indicating that the interrupt handling has been completed. (A write to 
the EOI register must not be included in the handler routine for an NMI, SMI, INIT, ExtINT, or SIPI.) 


10.8.2 Interrupt Handling with the P6 Family and Pentium Processors 


With the P6 family and Pentium processors, the local APIC handles the local interrupts, interrupt messages, and 
IPIs it receives as follows (see Figure 10-17). 


1. (IPIs only) The local APIC examines the IPI message to determines if it is the specified destination for the IPI 
as described in Section 10.6.2, “Determining IPI Destination.” If it is the specified destination, it continues its 
acceptance procedure; if it is not the destination, it discards the IPI message. When the message specifies 
lowest-priority delivery mode, the local APIC will arbitrate with the other processors that were designated as 
recipients of the IPI message (see Section 10.6.2.4, “Lowest Priority Delivery Mode”). 


2. If the local APIC determines that it is the designated destination for the interrupt and if the interrupt request is 
an NMI, SMI, INIT, ExtINT, or INIT-deassert interrupt, or one of the MP protocol IPI messages (BIPI, FIPI, and 
SIPI), the interrupt is sent directly to the processor core for handling. 


3. If the local APIC determines that it is the designated destination for the interrupt but the interrupt request is 
not one of the interrupts given in step 2, the local APIC looks for an open slot in one of its two pending interrupt 
queues contained in the IRR and ISR registers (see Figure 10-20). If a slot is available (see Section 10.8.4, 
“Interrupt Acceptance for Fixed Interrupts”), places the interrupt in the slot. If a slot is not available, it rejects 
the interrupt request and sends it back to the sender with a retry message. 


4. When interrupts are pending in the IRR register, the local APIC dispatches them to the processor one at a time, 
based on their priority and the current processor priority in the PPR (see Section 10.8.3.1, “Task and Processor 
Priorities”). 


5. When a fixed interrupt has been dispatched to the processor core for handling, the completion of the handler 
routine is indicated with an instruction in the instruction handler code that writes to the end-of-interrupt (EOI) 
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Figure 10-17. Interrupt Acceptance Flow Chart for the Local APIC (P6 Family and Pentium Processors) 


register in the local APIC (see Section 10.8.5, “Signaling Interrupt Servicing Completion”). The act of writing to 
the EOI register causes the local APIC to delete the interrupt from its queue and (for level-triggered interrupts) 
send a message on the bus indicating that the interrupt handling has been completed. (A write to the EOI 
register must not be included in the handler routine for an NMI, SMI, INIT, ExtINT, or SIPI.) 


The following sections describe the acceptance of interrupts and their handling by the local APIC and processor in 
greater detail. 


10.8.3 Interrupt, Task, and Processor Priority 


Each interrupt delivered to the processor through the local APIC has a priority based on its vector number. The local 
APIC uses this priority to determine when to service the interrupt relative to the other activities of the processor, 
including the servicing of other interrupts. 


Each interrupt vector is an 8-bit value. The interrupt-priority class is the value of bits 7:4 of the interrupt vector. 
The lowest interrupt-priority class is 1 and the highest is 15; interrupts with vectors in the range 0-15 (with inter- 
rupt-priority class 0) are illegal and are never delivered. Because vectors 0-31 are reserved for dedicated uses by 
the Intel 64 and IA-32 architectures, software should configure interrupt vectors to use interrupt-priority classes in 
the range 2-15. 


Each interrupt-priority class encompasses 16 vectors. The relative priority of interrupts within an interrupt-priority 
class is determined by the value of bits 3:0 of the vector number. The higher the value of those bits, the higher the 
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priority within that interrupt-priority class. Thus, each interrupt vector comprises two parts, with the high 4 bits 
indicating its interrupt-priority class and the low 4 bits indicating its ranking within the interrupt-priority class. 


10.8.3.1. Task and Processor Priorities 


The local APIC also defines a task priority and a processor priority that determine the order in which interrupts 
are handled. The task-priority class is the value of bits 7:4 of the task-priority register (TPR), which can be 
written by software (TPR is a read/write register); see Figure 10-18. 


31 87 43 0 


Reserved | | 


Task-Priority Class —————_—__ 
Address: FEEO 0080H Task-Priority Sub-Class 


Value after reset: OH 


Figure 10-18. Task-Priority Register (TPR) 


NOTE 


In this discussion, the term “task” refers to a software defined task, process, thread, program, or 
routine that is dispatched to run on the processor by the operating system. It does not refer to an 
IA-32 architecture defined task as described in Chapter 7, “Task Management.” 


The task priority allows software to set a priority threshold for interrupting the processor. This mechanism enables 
the operating system to temporarily block low priority interrupts from disturbing high-priority work that the 
processor is doing. The ability to block such interrupts using task priority results from the way that the TPR controls 
the value of the processor-priority register (PPR).° 


The processor-priority class is a value in the range 0-15 that is maintained in bits 7:4 of the processor-priority 
register (PPR); see Figure 10-19. The PPR is a read-only register. The processor-priority class represents the 
current priority at which the processor is executing. 


31 87 43 0 


Reserved | 


Processor-Priority Class 


Address: FEEO OOAOH Processor-Priority Sub-Class | 
Value after reset: OH 


Figure 10-19. Processor-Priority Register (PPR) 


The value of the PPR is based on the value of TPR and the value ISRV; ISRV is the vector number of the highest 
priority bit that is set in the ISR or OOH if no bit is set in the ISR. (See Section 10.8.4 for more details on the ISR.) 
The value of PPR is determined as follows: 


¢ PPR[7:4] (the processor-priority class) the maximum of TPR[7:4] (the task- priority class) and ISRV[7:4] (the 
priority of the highest priority interrupt in service). 


¢ PPR[3:0] (the processor-priority sub-class) is determined as follows: 
— If TPR[7:4] > ISRV[7:4], PPR[3:0] is TPR[3:0] (the task-priority sub-class). 
— If TPR[7:4] < ISRV[7:4], PPR[3:0] is 0. 
— If TPR[7:4] = ISRV[7:4], PPR[3:0] may be either TPR[3:0] or 0. The actual behavior is model-specific. 


6. The TPR also determines the arbitration priority of the local processor; see Section 10.6.2.4, “Lowest Priority Delivery Mode.” 
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The processor-priority class determines the priority threshold for interrupting the processor. The processor will 
deliver only those interrupts that have an interrupt-priority class higher than the processor-priority class in the 
PPR. If the processor-priority class is 0, the PPR does not inhibit the delivery any interrupt; if it is 15, the processor 
inhibits the delivery of all interrupts. (The processor-priority mechanism does not affect the delivery of interrupts 
with the NMI, SMI, INIT, ExtINT, INIT-deassert, and start-up delivery modes.) 


The processor does not use the processor-priority sub-class to determine which interrupts to delivery and which to 
inhibit. (The processor uses the processor-priority sub-class only to satisfy reads of the PPR.) 


10.8.4 Interrupt Acceptance for Fixed Interrupts 


The local APIC queues the fixed interrupts that it accepts in one of two interrupt pending registers: the interrupt 
request register (IRR) or in-service register (ISR). These two 256-bit read-only registers are shown in 

Figure 10-20. The 256 bits in these registers represent the 256 possible vectors; vectors 0 through 15 are reserved 
by the APIC (see also: Section 10.5.2, “Valid Interrupt Vectors”). 


NOTE 


All interrupts with an NMI, SMI, INIT, ExtINT, start-up, or INIT-deassert delivery mode bypass the 
IRR and ISR registers and are sent directly to the processor core for servicing. 


255 16 15 0 
Reserved IRR 


Reserved ISR 
Reserved TMR 


Addresses: IRR FEEO 0200H - FEEO 0270H 
ISR FEEO 0100H - FEEO 0170H 
TMR FEEO 0180H - FEEO 01FOH 
Value after reset: OH 


Figure 10-20. IRR, ISR and TMR Registers 


The IRR contains the active interrupt requests that have been accepted, but not yet dispatched to the processor for 
servicing. When the local APIC accepts an interrupt, it sets the bit in the IRR that corresponds the vector of the 
accepted interrupt. When the processor core is ready to handle the next interrupt, the local APIC clears the highest 
priority IRR bit that is set and sets the corresponding ISR bit. The vector for the highest priority bit set in the ISR 
is then dispatched to the processor core for servicing. 


While the processor is servicing the highest priority interrupt, the local APIC can send additional fixed interrupts by 
setting bits in the IRR. When the interrupt service routine issues a write to the EOI register (see Section 10.8.5, 
“Signaling Interrupt Servicing Completion”), the local APIC responds by clearing the highest priority ISR bit that is 
set. It then repeats the process of clearing the highest priority bit in the IRR and setting the corresponding bit in 
the ISR. The processor core then begins executing the service routing for the highest priority bit set in the ISR. 


If more than one interrupt is generated with the same vector number, the local APIC can set the bit for the vector 
both in the IRR and the ISR. This means that for the Pentium 4 and Intel Xeon processors, the IRR and ISR can 
queue two interrupts for each interrupt vector: one in the IRR and one in the ISR. Any additional interrupts issued 
for the same interrupt vector are collapsed into the single bit in the IRR. 


For the P6 family and Pentium processors, the IRR and ISR registers can queue no more than two interrupts per 
interrupt vector and will reject other interrupts that are received within the same vector. 


If the local APIC receives an interrupt with an interrupt-priority class higher than that of the interrupt currently in 
service, and interrupts are enabled in the processor core, the local APIC dispatches the higher priority interrupt to 
the processor immediately (without waiting for a write to the EOI register). The currently executing interrupt 
handler is then interrupted so the higher-priority interrupt can be handled. When the handling of the higher- 
priority interrupt has been completed, the servicing of the interrupted interrupt is resumed. 
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The trigger mode register (TMR) indicates the trigger mode of the interrupt (see Figure 10-20). Upon acceptance 
of an interrupt into the IRR, the corresponding TMR bit is cleared for edge-triggered interrupts and set for level- 
triggered interrupts. If a TMR bit is set when an EOI cycle for its corresponding interrupt vector is generated, an EOI 
message is sent to all I/O APICs. 


10.8.5 Signaling Interrupt Servicing Completion 


For all interrupts except those delivered with the NMI, SMI, INIT, ExtINT, the start-up, or INIT-Deassert delivery 
mode, the interrupt handler must include a write to the end-of-interrupt (EOI) register (see Figure 10-21). This 
write must occur at the end of the handler routine, sometime before the IRET instruction. This action indicates that 
the servicing of the current interrupt is complete and the local APIC can issue the next interrupt from the ISR. 


31 0 


Address: OFEEO 00OBOH 
Value after reset: OH 


Figure 10-21. EOI Register 


Upon receiving an EOI, the APIC clears the highest priority bit in the ISR and dispatches the next highest priority 
interrupt to the processor. If the terminated interrupt was a level-triggered interrupt, the local APIC also sends an 
end-of-interrupt message to all I/O APICs. 


System software may prefer to direct EOIs to specific I/O APICs rather than having the local APIC send end-of- 
interrupt messages to all I/O APICs. 


Software can inhibit the broadcast of EOI message by setting bit 12 of the Spurious Interrupt Vector Register (see 
Section 10.9). If this bit is set, a broadcast EOI is not generated on an EOI cycle even if the associated TMR bit indi- 
cates that the current interrupt was level-triggered. The default value for the bit is 0, indicating that EOI broadcasts 
are performed. 


Bit 12 of the Spurious Interrupt Vector Register is reserved to 0 if the processor does not support suppression of 
EOI broadcasts. Support for EOI-broadcast suppression is reported in bit 24 in the Local APIC Version Register (see 
Section 10.4.8); the feature is supported if that bit is set to 1. When supported, the feature is available in both 
xAPIC mode and x2APIC mode. 


System software desiring to perform directed EOIs for level-triggered interrupts should set bit 12 of the Spurious 
Interrupt Vector Register and follow each the EOI to the local xAPIC for a level triggered interrupt with a directed 
EOI to the I/O APIC generating the interrupt (this is done by writing to the I/O APIC’s EOI register). System soft- 
ware performing directed EOIs must retain a mapping associating level-triggered interrupts with the I/O APICs in 
the system. 


10.8.6 Task Priority in |A-32e Mode 


In IA-32e mode, operating systems can manage the 16 interrupt-priority classes (see Section 10.8.3, “Interrupt, 

Task, and Processor Priority”) explicitly using the task priority register (TPR). Operating systems can use the TPR 

to temporarily block specific (low-priority) interrupts from interrupting a high-priority task. This is done by loading 

TPR with a value in which the task-priority class corresponds to the highest interrupt-priority class that is to be 

blocked. For example: 

¢ Loading the TPR with a task-priority class of 8 (01000B) blocks all interrupts with an interrupt-priority class of 
8 or less while allowing all interrupts with an interrupt-priority class of 9 or more to be recognized. 

¢ Loading the TPR with a task-priority class of 0 enables all external interrupts. 

¢ Loading the TPR with a task-priority class of OFH (01111B) disables all external interrupts. 


The TPR (shown in Figure 10-18) is cleared to 0 on reset. In 64-bit mode, software can read and write the TPR 
using an alternate interface, MOV CR8 instruction. The new task-priority class is established when the MOV CR8 
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instruction completes execution. Software does not need to force serialization after loading the TPR using MOV 
CR8. 


Use of the MOV CRn instruction requires a privilege level of 0. Programs running at privilege level greater than 0 
cannot read or write the TPR. An attempt to do so causes a general-protection exception. The TPR is abstracted 
from the interrupt controller (IC), which prioritizes and manages external interrupt delivery to the processor. The 
IC can be an external device, such as an APIC or 8259. Typically, the IC provides a priority mechanism similar or 
identical to the TPR. The IC, however, is considered implementation-dependent with the under-lying priority mech- 
anisms subject to change. CR8, by contrast, is part of the Intel 64 architecture. Software can depend on this defi- 
nition remaining unchanged. 


Figure 10-22 shows the layout of CR8; only the low four bits are used. The remaining 60 bits are reserved and must 
be written with zeros. Failure to do this causes a general-protection exception. 


63 43 0 


Reserved 


Value after reset: OH 


Figure 10-22. CR8 Register 


10.8.6.1 Interaction of Task Priorities between CR8 and APIC 


The first implementation of Intel 64 architecture includes a local advanced programmable interrupt controller 
(APIC) that is similar to the APIC used with previous IA-32 processors. Some aspects of the local APIC affect the 
operation of the architecturally defined task priority register and the programming interface using CR8. 


Notable CR8 and APIC interactions are: 

¢ The processor powers up with the local APIC enabled. 

¢ The APIC must be enabled for CR8 to function as the TPR. Writes to CR8 are reflected into the APIC Task Priority 
Register. 

¢ =©APIC.TPR[bits 7:4] = CR8[bits 3:0], APIC.TPR[bits 3:0] = 0. A read of CR8 returns a 64-bit value which is the 
value of TPR[bits 7:4], zero extended to 64 bits. 


There are no ordering mechanisms between direct updates of the APIC.TPR and CR8. Operating software should 
implement either direct APIC TPR updates or CR8 style TPR updates but not mix them. Software can use a serial- 
izing instruction (for example, CPUID) to serialize updates between MOV CR8 and stores to the APIC. 


10.9 SPURIOUS INTERRUPT 


A special situation may occur when a processor raises its task priority to be greater than or equal to the level of the 
interrupt for which the processor INTR signal is currently being asserted. If at the time the INTA cycle is issued, the 
interrupt that was to be dispensed has become masked (programmed by software), the local APIC will deliver a 
spurious-interrupt vector. Dispensing the spurious-interrupt vector does not affect the ISR, so the handler for this 
vector should return without an EOI. 


The vector number for the spurious-interrupt vector is specified in the spurious-interrupt vector register (see 

Figure 10-23). The functions of the fields in this register are as follows: 

Spurious Vector Determines the vector number to be delivered to the processor when the local APIC generates 
a spurious vector. 
(Pentium 4 and Intel Xeon processors.) Bits 0 through 7 of the this field are programmable by 
software. 


(P6 family and Pentium processors). Bits 4 through 7 of the this field are programmable by 
software, and bits 0 through 3 are hardwired to logical ones. Software writes to bits 0 through 
3 have no effect. 


APIC Software Enable/ Disable 
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Allows software to temporarily enable (1) or disable (0) the local APIC (see Section 10.4.3, 
“Enabling or Disabling the Local APIC”). 


Focus Processor Checking 
Determines if focus processor checking is enabled (0) or disabled (1) when using the lowest- 
priority delivery mode. In Pentium 4 and Intel Xeon processors, this bit is reserved and should 
be cleared to 0. 

Suppress EOI Broadcasts 


Determines whether an EOI for a level-triggered interrupt causes EOI messages to be broad- 
cast to the I/O APICs (0) or not (1). See Section 10.8.5. The default value for this bit is 0, indi- 
cating that EOI broadcasts are performed. This bit is reserved to 0 if the processor does not 
support EOI-broadcast suppression. 


NOTE 


Do not program an LVT or IOAPIC RTE with a spurious vector even if you set the mask bit. A 
spurious vector ISR does not do an EOI. If for some reason an interrupt is generated by an LVT or 
RTE entry, the bit in the in-service register will be left set for the spurious vector. This will mask all 
interrupts at the same or lower priority 


31 121110 9 8 7 0 


1 


EOI-Broadcast Suppression 
0: Disabled 
1: Enabled 


Focus Processor Checking2 
0: Enabled 
1: Disabled 


APIC Software Enable/Disable 
0: APIC Disabled 
1: APIC Enabled 


Spurious Vector? 


Address: FEEO OOFOH 
Value after reset: 0000 OOFFH 


1. Not supported on all processors. See bit 24 of Local APIC Version Register. 
2. Not supported in Pentium 4 and Intel Xeon processors. 
3. For the P6 family and Pentium processors, bits 0 through 3 

are always 0. 


Figure 10-23. Spurious-Interrupt Vector Register (SVR) 


10.10 APIC BUS MESSAGE PASSING MECHANISM AND 
PROTOCOL (P6 FAMILY, PENTIUM PROCESSORS) 


The Pentium 4 and Intel Xeon processors pass messages among the local and I/O APICs on the system bus, using 
the system bus message passing mechanism and protocol. 


The P6 family and Pentium processors, pass messages among the local and I/O APICs on the serial APIC bus, as 
follows. Because only one message can be sent at a time on the APIC bus, the I/O APIC and local APICs employ a 
‘rotating priority” arbitration protocol to gain permission to send a message on the APIC bus. One or more APICs 
may start sending their messages simultaneously. At the beginning of every message, each APIC presents the type 
of the message it is sending and its current arbitration priority on the APIC bus. This information is used for arbi- 
tration. After each arbitration cycle (within an arbitration round), only the potential winners keep driving the bus. 
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By the time all arbitration cycles are completed, there will be only one APIC left driving the bus. Once a winner is 
selected, it is granted exclusive use of the bus, and will continue driving the bus to send its actual message. 


After each successfully transmitted message, all APICs increase their arbitration priority by 1. The previous winner 
(that is, the one that has just successfully transmitted its message) assumes a priority of 0 (lowest). An agent 
whose arbitration priority was 15 (highest) during arbitration, but did not send a message, adopts the previous 
winner's arbitration priority, incremented by 1. 


Note that the arbitration protocol described above is slightly different if one of the APICs issues a special End-Of- 
Interrupt (EOI). This high-priority message is granted the bus regardless of its sender’s arbitration priority, unless 
more than one APIC issues an EOI message simultaneously. In the latter case, the APICs sending the EOI 
messages arbitrate using their arbitration priorities. 


If the APICs are set up to use “lowest priority” arbitration (see Section 10.6.2.4, “Lowest Priority Delivery Mode”) 
and multiple APICs are currently executing at the lowest priority (the value in the APR register), the arbitration 
priorities (unique values in the Arb ID register) are used to break ties. All 8 bits of the APR are used for the lowest 
priority arbitration. 


10.10.1 Bus Message Formats 


See Section 10.13, “APIC Bus Message Formats,” for a description of bus message formats used to transmit 
messages on the serial APIC bus. 


10.11 MESSAGE SIGNALLED INTERRUPTS 


The PCI Local Bus Specification, Rev 2.2 (www.pcisig.com) introduces the concept of message signalled interrupts. 
As the specification indicates: 


“Message signalled interrupts (MSI) is an optional feature that enables PCI devices to request 
service by writing a system-specified message to a system-specified address (PCI DWORD memory 
write transaction). The transaction address specifies the message destination while the transaction 
data specifies the message. System software is expected to initialize the message destination and 
message during device configuration, allocating one or more non-shared messages to each MSI 
capable function.” 


The capabilities mechanism provided by the PCI Local Bus Specification is used to identify and configure MSI 
capable PCI devices. Among other fields, this structure contains a Message Data Register and a Message Address 
Register. To request service, the PCI device function writes the contents of the Message Data Register to the 
address contained in the Message Address Register (and the Message Upper Address register for 64-bit message 
addresses). 


Section 10.11.1 and Section 10.11.2 provide layout details for the Message Address Register and the Message Data 
Register. The operation issued by the device is a PCI write command to the Message Address Register with the 
Message Data Register contents. The operation follows semantic rules as defined for PCI write operations and is a 
DWORD operation. 


10.11.1 Message Address Register Format 
The format of the Message Address Register (lower 32-bits) is shown in Figure 10-24. 


31 20 19 12 11 4 3 2 1 0 


OFEEH Destination ID | Reserved RH DM XX 


Figure 10-24. Layout of the MSI Message Address Register 
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Fields in the Message Address Register are as follows: 


1. 


4. 


Bits 31-20 — These bits contain a fixed value for interrupt messages (OFEEH). This value locates interrupts at 
the 1-MByte area with a base address of 4G - 18M. All accesses to this region are directed as interrupt 
messages. Care must to be taken to ensure that no other device claims the region as I/O space. 


Destination ID — This field contains an 8-bit destination ID. It identifies the message’s target processor(s). 
The destination ID corresponds to bits 63:56 of the I/O APIC Redirection Table Entry if the IOAPIC is used to 
dispatch the interrupt to the processor(s). 


Redirection hint indication (RH) — When this bit is set, the message is directed to the processor with the 
lowest interrupt priority among processors that can receive the interrupt. 


¢ When RH is 0, the interrupt is directed to the processor listed in the Destination ID field. 


¢ When RH is 1 and the physical destination mode is used, the Destination ID field must not be set to FFH; 
it must point to a processor that is present and enabled to receive the interrupt. 


¢ When RH is 1 and the logical destination mode is active in a system using a flat addressing model, the 
Destination ID field must be set so that bits set to 1 identify processors that are present and enabled to 
receive the interrupt. 


¢ If RH is set to 1 and the logical destination mode is active in a system using cluster addressing model, 
then Destination ID field must not be set to FFH; the processors identified with this field must be 
present and enabled to receive the interrupt. 


Destination mode (DM) — This bit indicates whether the Destination ID field should be interpreted as logical 
or physical APIC ID for delivery of the lowest priority interrupt. 


¢ If RHis 1 and DM is 0, the Destination ID field is in physical destination mode and only the processor in 
the system that has the matching APIC ID is considered for delivery of that interrupt (this means no re- 
direction). 


¢ If RH is 1 and DM is 1, the Destination ID Field is interpreted as in logical destination mode and the 
redirection is limited to only those processors that are part of the logical group of processors based on 
the processor’s logical APIC ID and the Destination ID field in the message. The logical group of 
processors consists of those identified by matching the 8-bit Destination ID with the logical destination 
identified by the Destination Format Register and the Logical Destination Register in each local APIC. 
The details are similar to those described in Section 10.6.2, “Determining IPI Destination.” 


¢ If RH is 0, then the DM bit is ignored and the message is sent ahead independent of whether the 
physical or logical destination mode is used. 


10.11.2 Message Data Register Format 


The layout of the Message Data Register is shown in Figure 10-25. 


Reserved fields are not assumed to be any value. Software must preserve their contents on writes. Other fields in 
the Message Data Register are described below. 


1. 


Vector — This 8-bit field contains the interrupt vector associated with the message. Values range from 010H 
to OFEH. Software must guarantee that the field is not programmed with vector OOH to OFH. 


Delivery Mode — This 3-bit field specifies how the interrupt receipt is handled. Delivery Modes operate only in 
conjunction with specified Trigger Modes. Correct Trigger Modes must be guaranteed by software. Restrictions 
are indicated below: 


a. OOOB (Fixed Mode) — Deliver the signal to all the agents listed in the destination. The Trigger Mode for 
fixed delivery mode can be edge or level. 


b. 001B (Lowest Priority) — Deliver the signal to the agent that is executing at the lowest priority of all 
agents listed in the destination field. The trigger mode can be edge or level. 


c. 010B (System Management Interrupt or SMI) — The delivery mode is edge only. For systems that rely 
on SMI semantics, the vector field is ignored but must be programmed to all zeroes for future compatibility. 
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63 32 
Reserved 
31 16 15 14. 13 11 10 8 7 0 
Reserved Reserved Vector 
Trigger Mode Delivery Mode 
0 - Edge 000 - Fixed 
1 - Level 001 - Lowest Priority 
J 010 - SMI 
Level for Trigger Mode = 0 F35 ; ee 
X - Don’t care On 
Level for Trigger Mode = 1 101 - INIT 
0 - Deassert 110 - Reserved 
1 - Assert 111 - ExtINT 


Figure 10-25. Layout of the MSI Message Data Register 


d. 100B (NMI) — Deliver the signal to all the agents listed in the destination field. The vector information is 
ignored. NMI is an edge triggered interrupt regardless of the Trigger Mode Setting. 


e. 101B (INIT) — Deliver this signal to all the agents listed in the destination field. The vector information is 
ignored. INIT is an edge triggered interrupt regardless of the Trigger Mode Setting. 


f. 111B (ExtINT) — Deliver the signal to the INTR signal of all agents in the destination field (as an interrupt 
that originated from an 8259A compatible interrupt controller). The vector is supplied by the INTA cycle 
issued by the activation of the ExtINT. ExtINT is an edge triggered interrupt. 


3. Level — Edge triggered interrupt messages are always interpreted as assert messages. For edge triggered 
interrupts this field is not used. For level triggered interrupts, this bit reflects the state of the interrupt input. 


4. Trigger Mode — This field indicates the signal type that will trigger a message. 
a. O — Indicates edge sensitive. 
b. 1 -— Indicates level sensitive. 


10.12 EXTENDED XAPIC (X2APIC) 


The x2APIC architecture extends the xAPIC architecture (described in Section 10.4) in a backward compatible 
manner and provides forward extendability for future Intel platform innovations. Specifically, the x2APIC architec- 
ture does the following. 


¢ Retains all key elements of compatibility to the xAPIC architecture. 
— Delivery modes. 
— Interrupt and processor priorities. 
— Interrupt sources. 
— Interrupt destination types. 
¢ Provides extensions to scale processor addressability for both the logical and physical destination modes. 


Vol. 3A 10-37 


ADVANCED PROGRAMMABLE INTERRUPT CONTROLLER (APIC) 


e Adds new features to enhance performance of interrupt delivery. 
¢ Reduces complexity of logical destination mode interrupt delivery on link based platform architectures. 


¢ Uses MSR programming interface to access APIC registers in x2APIC mode instead of memory-mapped 
interfaces. Memory-mapped interface is supported when operating in xAPIC mode. 


10.12.1 Detecting and Enabling x2APIC Mode 


Processor support for x2APIC mode can be detected by executing CPUID with EAX=1 and then checking ECX, bit 21 
ECX. If CPUID.(EAX=1):ECX.21 is set , the processor supports the x2APIC capability and can be placed into the 
x2APIC mode. 


System software can place the local APIC in the x2APIC mode by setting the x2APIC mode enable bit (bit 10) in the 
IA32_APIC_BASE MSR at MSR address 01BH. The layout for the IA32_APIC_BASE MSR is shown in Figure 10-26. 


63 36 35 121110987 0 


Reserved APIC Base | | | 


APIC Base—Base physical address 
EN—xAPIC global enable/disable 
EXTD—Enable x2APIC mode 

BSP—Processor is BSP 


Reserved 


Figure 10-26. IA32_APIC_BASE MSR Supporting x2APIC 


Table 10-5, “x2APIC operating mode configurations” describe the possible combinations of the enable bit (EN - bit 
11) and the extended mode bit (EXTD - bit 10) in the IA32_APIC_BASE MSR. 


Table 10-5. x2APIC Operating Mode Configurations 


XAPIC global enable x2APIC enable 
(IA32_APIC_BASE[11]) (IA32_APIC_BASE[10]) Description 
0 0 local APIC is disabled 
0 1 Invalid 
1 0 local APIC is enabled in xAPIC mode 
1 1 local APIC is enabled in x2APIC mode 


Once the local APIC has been switched to x2APIC mode (EN = 1, EXTD = 1), switching back to xAPIC mode would 
require system software to disable the local APIC unit. Specifically, attempting to write a value to the 
IA32_APIC_BASE MSR that has (EN= 1, EXTD = 0) when the local APIC is enabled and in x2APIC mode causes a 
general-protection exception. Once bit 10 in IA32_APIC_BASE MSR is set, the only way to leave x2APIC mode using 
IA32_APIC_BASE would require a WRMSR to set both bit 11 and bit 10 to zero. Section 10.12.5, “x2APIC State 
Transitions” provides a detailed state diagram for the state transitions allowed for the local APIC. 


10.12.1.1 Instructions to Access APIC Registers 


In x2APIC mode, system software uses RDMSR and WRMSR to access the APIC registers. The MSR addresses for 
accessing the x2APIC registers are architecturally defined and specified in Section 10.12.1.2, “x2APIC Register 
Address Space”. Executing the RDMSR instruction with the APIC register address specified in ECX returns the 
content of bits 0 through 31 of the APIC registers in EAX. Bits 32 through 63 are returned in register EDX - these 
bits are reserved if the APIC register being read is a 32-bit register. Similarly executing the WRMSR instruction with 
the APIC register address in ECX, writes bits 0 to 31 of register EAX to bits 0 to 31 of the specified APIC register. If 
the register is a 64-bit register then bits 0 to 31 of register EDX are written to bits 32 to 63 of the APIC register. The 
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Interrupt Command Register is the only APIC register that is implemented as a 64-bit MSR. The semantics of 
handling reserved bits are defined in Section 10.12.1.3, “Reserved Bit Checking”. 


10.12.1.2 x2APIC Register Address Space 


The MSR address range 800H through 8FFH is architecturally reserved and dedicated for accessing APIC registers 
in x2APIC mode. Table 10-6 lists the APIC registers that are available in x2APIC mode. When appropriate, the table 
also gives the offset at which each register is available on the page referenced by IA32_APIC_BASE[35:12] in 
xAPIC mode. 


There is a one-to-one mapping between the x2APIC MSRs and the legacy xAPIC register offsets with the following 

exceptions: 

¢ The Destination Format Register (DFR): The DFR, supported at offset OEQH in xAPIC mode, is not supported in 
x2APIC mode. There is no MSR with address 80EH. 


° The Interrupt Command Register (ICR): The two 32-bit registers in xAPIC mode (at offsets 300H and 310H) are 
merged into a single 64-bit MSR in x2APIC mode (with MSR address 830H). There is no MSR with address 
831H. 

¢ The SELF IPI register. This register is available only in x2APIC mode at address 83FH. In xAPIC mode, there is 
no register defined at offset 3FOH. 


MSR addresses in the range 800H-8FFH that are not listed in Table 10-6 (including 80EH and 831H) are reserved. 
Executions of RDMSR and WRMSR that attempt to access such addresses cause general-protection exceptions. 


The MSR address space is compressed to allow for future growth. Every 32 bit register on a 128-bit boundary in the 
legacy MMIO space is mapped to a single MSR in the local x2APIC MSR address space. The upper 32-bits of all 
x2APIC MSRs (except for the ICR) are reserved. 


Table 10-6. Local APIC Register Address Map Supported by x2APIC 


MSR Address MMIO Offset Reaictar Name MSR R/W Comments 
(x2APIC mode) (xAPIC mode) g Semantics 
802H 020H Local APIC ID register Read-only! See Section 10.12.5.1 for initial 
values. 
803H 030H Local APIC Version register Read-only Same version used in xAPIC mode 
and x2APIC mode. 
808H O80H Task Priority Register (TPR) Read/write Bits 31:8 are reserved. 
80AH OAOH Processor Priority Register Read-only 
(PPR) 
80BH OBOH EOI register Write-only? WRMSR of a non-zero value causes 
#GP(0). 
80DH ODOH Logical Destination Register Read-only Read/write in xAPIC mode. 
(LDR) 
80FH OFOH Spurious Interrupt Vector Read/write See Section 10.9 for reserved bits. 
Register (SVR) 
810H 100H In-Service Register (ISR); bits | Read-only 
31:0 
811H 110H ISR bits 63:32 Read-only 
812H 120H ISR bits 95:64 Read-only 
813H 130H ISR bits 127:96 Read-only 
814H 140H ISR bits 159:128 Read-only 
815H 150H ISR bits 191:160 Read-only 
816H 160H ISR bits 223:192 Read-only 
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Table 10-6. Local APIC Register Address Map Supported by x2APIC (Contd.) 


MSR Address MMIO Offset Register Name MSR R/W Comments 
(x2APIC mode) (xAPIC mode) Semantics 
817H 170H ISR bits 255:224 Read-only 
818H 180H Trigger Mode Register (TMR); | Read-only 
bits 31:0 
819H 190H TMR bits 63:32 Read-only 
81AH 1A0H TMR bits 95:64 Read-only 
81BH 1BOH TMR bits 127:96 Read-only 
81CH 1COH TMR bits 159:128 Read-only 
81DH 1DOH TMR bits 191:160 Read-only 
81EH 1€0H TMR bits 223:192 Read-only 
81FH 1FOH TMR bits 255:224 Read-only 
820H 200H Interrupt Request Register Read-only 
(IRR); bits 31:0 
821H 210H IRR bits 63:32 Read-only 
822H 220H IRR bits 95:64 Read-only 
823H 230H IRR bits 127:96 Read-only 
824H 240H IRR bits 159:128 Read-only 
825H 250H IRR bits 191:160 Read-only 
826H 260H IRR bits 223:192 Read-only 
827H 270H IRR bits 255:224 Read-only 
828H 280H Error Status Register (ESR) Read/write WRMSR of a non-zero value causes 
#GP(0). See Section 10.5.3. 
82FH 2FOH LWT CMCI register Read/write See Figure 10-8 for reserved bits. 
830H* 300H and 310H Interrupt Command Register Read/write See Figure 10-28 for reserved bits 
(ICR) 
832H 320H LVT Timer register Read/write See Figure 10-8 for reserved bits. 
833H 330H LVT Thermal Sensor register | Read/write See Figure 10-8 for reserved bits. 
834H 340H LVT Performance Monitoring | Read/write See Figure 10-8 for reserved bits. 
register 
835H 350H LVT LINTO register Read/write See Figure 10-8 for reserved bits. 
836H 360H LWT LINT1 register Read/write See Figure 10-8 for reserved bits. 
837H 370H LT Error register Read/write See Figure 10-8 for reserved bits. 
838H 380H Initial Count register (for Read/write 
Timer) 
839H 390H Current Count register (for Read-only 
Timer) 
83E€H 3€0H Divide Configuration Register | Read/write See Figure 10-10 for reserved bits. 
(DCR; for Timer) 
83FH Not available SELF IPI° Write-only Available only in x2APIC mode. 
NOTES: 


1. WRMSR causes #GP(0) for read-only registers. 
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2. WRMSR causes #GP(0) for attempts to set a reserved bit to 1 in a read/write register (including bits 63:32 of each register). 
3. RDMSR causes #GP(0) for write-only registers. 


4. MSR 831H is reserved; read/write operations cause general-protection exceptions. The contents of the APIC register at MMIO offset 
310H are accessible in x2APIC mode through the MSR at address 830H. 


5. SELF IPI register is supported only in x2APIC mode. 


10.12.1.3 Reserved Bit Checking 


Section 10.12.1.2 and Table 10-6 specifies the reserved bit definitions for the APIC registers in x2APIC mode. Non- 
zero writes (by WRMSR instruction) to reserved bits to these registers will raise a general protection fault exception 
while reads return zeros (RsvdZ semantics). 


In x2APIC mode, the local APIC ID register is increased to 32 bits wide. This enables -1 processors to be 
addressable in physical destination mode. This 32-bit value is referred to as “x2APIC ID”. A processor implementa- 
tion may choose to support less than 32 bits in its hardware. System software should be agnostic to the actual 
number of bits that are implemented. All non-implemented bits will return zeros on reads by software. 


The APIC ID value of FFFF_FFFFH and the highest value corresponding to the implemented bit-width of the local 
APIC ID register in the system are reserved and cannot be assigned to any logical processor. 


232 


In x2APIC mode, the local APIC ID register is a read-only register to system software and will be initialized by hard- 
ware. It is accessed via the RDMSR instruction reading the MSR at address 0802H. 


Each logical processor in the system (including clusters with a communication fabric) must be configured with an 
unique x2APIC ID to avoid collisions of x2APIC IDs. On DP and high-end MP processors targeted to specific market 
segments and depending on the system configuration, it is possible that logical processors in different and “un- 
connected” clusters power up initialized with overlapping x2APIC IDs. In these configurations, a model-specific 
means may be provided in those product segments to enable BIOS and/or platform firmware to re-configure the 
x2APIC IDs in some clusters to provide for unique and non-overlapping system wide IDs before configuring the 
disconnected components into a single system. 


10.12.2 x2APIC Register Availability 


The local APIC registers can be accessed via the MSR interface only when the local APIC has been switched to the 
x2APIC mode as described in Section 10.12.1. Accessing any APIC register in the MSR address range 0800H 
through O8FFH via RDMSR or WRMSR when the local APIC is not in x2APIC mode causes a general-protection 
exception. In x2APIC mode, the memory mapped interface is not available and any access to the MMIO interface 
will behave similar to that of a legacy xAPIC in globally disabled state. Table 10-7 provides the interactions between 
the legacy & extended modes and the legacy and register interfaces. 


Table 10-7. MSR/MMIO Interface of a Local x2APIC in Different Modes of Operation 


MMIO Interface MSR Interface 
xAPIC mode Available General-protection exception 
x2APIC mode Behavior identical to xAPIC in globally disabled state Available 


10.12.3. MSR Access in x2APIC Mode 


To allow for efficient access to the APIC registers in x2APIC mode, the serializing semantics of WRMSR are relaxed 
when writing to the APIC registers. Thus, system software should not use “WRMSR to APIC registers in x2APIC 
mode” as a Serializing instruction. Read and write accesses to the APIC registers will occur in program order. A 
WRMSR to an APIC register may complete before all preceding stores are globally visible; software can prevent this 
by inserting a serializing instruction or the sequence MFENCE;LFENCE before the WRMSR. 


The RDMSR instruction is not serializing and this behavior is unchanged when reading APIC registers in x2APIC 
mode. System software accessing the APIC registers using the RDMSR instruction should not expect a serializing 
behavior. (Note: The MMIO-based xAPIC interface is mapped by system software as an un-cached region. Conse- 
quently, read/writes to the xAPIC-MMIO interface have serializing semantics in the xAPIC mode.) 
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10.12.4 VM-Exit Controls for MSRs and x2APIC Registers 


The VMX architecture allows a VMM to specify lists of MSRs to be loaded or stored on VMX transitions using the 
VMxX-transition MSR areas (see VM-exit MSR-store address field, VM-exit MSR-load address field, and VM-entry 
MSR-load address field in Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C). 


The X2APIC MSRs cannot to be loaded and stored on VMX transitions. A VMX transition fails if the VMM has specified 
that the transition should access any MSRs in the address range from 0000_0800H to 0000_O8FFH (the range used 
for accessing the X2APIC registers). Specifically, processing of an 128-bit entry in any of the VMX-transition MSR 
areas fails if bits 31:0 of that entry (represented as ENTRY_LOW_DW) satisfies the expression: “ENTRY_LOW_DW 
& FFFFF800H = O0000800H”. Such a failure causes an associated VM entry to fail (by reloading host state) and 
causes an associated VM exit to lead to VMX abort. 


10.12.5 x2APIC State Transitions 


This section provides a detailed description of the x2APIC states of a local x2APIC unit, transitions between these 
states as well as interactions of these states with INIT and reset. 


10.12.5.1 x2APIC States 

The valid states for a local x2APIC unit are listed in Table 10-5. 

e APIC disabled: IA32_APIC_BASE[EN]=0 and IA32_APIC_BASE[EXTD]=0. 
¢ xAPIC mode: IA32_APIC_BASE[EN]=1 and IA32_APIC_BASE[EXTD]=0. 
° x2APIC mode: IA32_APIC_BASE[EN]=1 and IA32_APIC_BASE[EXTD]=1. 
° Invalid: IA32_APIC_BASE[EN]=0 and IA32_APIC_BASE[EXTD]=1. 


The state corresponding to EXTD=1 and EN=0 is not valid and it is not possible to get into this state. An execution 
of WRMSR to the IA32_APIC_BASE_MSR that attempts a transition from a valid state to this invalid state causes a 
general-protection exception. Figure 10-27 shows the comprehensive state transition diagram for a local x2APIC 
unit. 


On coming out of reset, the local APIC unit is enabled and is in the xAPIC mode: IA32_APIC_BASE[EN]=1 and 
IA32_APIC_BASE[EXTD]=0. The APIC registers are initialized as follows. 


¢ The local APIC ID is initialized by hardware with a 32 bit ID (x2APIC ID). The lowest 8 bits of the x2APIC ID are 
the legacy local xAPIC ID, and are stored in the upper 8 bits of the APIC register for access in xAPIC mode. 


¢ The following APIC registers are reset to all zeros for those fields that are defined in the xAPIC mode. 


— IRR, ISR, TMR, ICR, LDR, TPR, Divide Configuration Register (See Section 10.4 through Section 10.6 for 
details of individual APIC registers). 


— Timer initial count and timer current count registers. 
° The LVT registers are reset to Os except for the mask bits; these are set to 1s. 
¢ The local APIC version register is not affected. 
¢ The Spurious Interrupt Vector Register is initialized to OOOOOOFFH. 
¢ The DFR (available only in xAPIC mode) is reset to all is. 
° SELF IPI register is reset to zero. 
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Figure 10-27. Local x2APIC State Transitions with IA32_APIC_BASE, INIT, and Reset 


Reset 


x2APIC After Reset 


The valid transitions from the xAPIC mode state are: 


¢ tothe x2APIC mode by setting EXT to 1 (resulting EN=1, EXTD= 1). The physical x2APIC ID (see Figure 10-6) 
is preserved across this transition and the logical x2APIC ID (see Figure 10-29) is initialized by hardware during 
this transition as documented in Section 10.12.10.2. The state of the extended fields in other APIC registers, 
which was not initialized at reset, is not architecturally defined across this transition and system software 
should explicitly initialize those programmable APIC registers. 


* to the disabled state by setting EN to 0 (resulting EN=0, EXTD= 0). 


The result of an INIT in the xAPIC state places the APIC in the state with EN= 1, EXTD= 0. The state of the local 
APIC ID register is preserved (the 8-bit xAPIC ID is in the upper 8 bits of the APIC ID register). All the other APIC 
registers are initialized as a result of INIT. 


A reset in this state places the APIC in the state with EN= 1, EXTD= O. The state of the local APIC ID register is 
initialized as described in Section 10.12.5.1. All the other APIC registers are initialized described in Section 
10.12.5.1. 


X2APIC Transitions From x2APIC Mode 


From the x2APIC mode, the only valid x2APIC transition using IA32_APIC_BASE is to the state where the x2APIC 
is disabled by setting EN to 0 and EXTD to 0. The x2APIC ID (32 bits) and the legacy local xAPIC ID (8 bits) are 
preserved across this transition. A transition from the x2APIC mode to xAPIC mode is not valid, and the corre- 
sponding WRMSR to the IA32_APIC_BASE MSR causes a general-protection exception. 

A reset in this state places the x2APIC in xAPIC mode. All APIC registers (including the local APIC ID register) are 
initialized as described in Section 10.12.5.1. 


An INIT in this state keeps the x2APIC in the x2APIC mode. The state of the local APIC ID register is preserved (all 
32 bits). However, all the other APIC registers are initialized as a result of the INIT transition. 
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X2APIC Transitions From Disabled Mode 


From the disabled state, the only valid x2APIC transition using IA32_APIC_BASE is to the xAPIC mode (EN= 1, 
EXTD = 0). Thus the only means to transition from x2APIC mode to xAPIC mode is a two-step process: 


° first transition from x2APIC mode to local APIC disabled mode (EN= 0, EXTD = 0), 
¢ followed by another transition from disabled mode to xAPIC mode (EN= 1, EXTD= 0). 


Consequently, all the APIC register states in the x2APIC, except for the x2APIC ID (32 bits), are not preserved 
across mode transitions. 


A reset in the disabled state places the x2APIC in the xAPIC mode. All APIC registers (including the local APIC ID 
register) are initialized as described in Section 10.12.5.1. 


An INIT in the disabled state keeps the x2APIC in the disabled state. 


State Changes From xAPIC Mode to x2APIC Mode 


After APIC register states have been initialized by software in xAPIC mode, a transition from xAPIC mode to x2APIC 
mode does not affect most of the APIC register states, except the following: 


°* The Logical Destination Register is not preserved. 
¢ Any APIC ID value written to the memory-mapped local APIC ID register is not preserved. 
¢ The high half of the Interrupt Command Register is not preserved. 


10.12.6 Routing of Device Interrupts in x2APIC Mode 


The x2APIC architecture is intended to work with all existing IOxAPIC units as well as all PCI and PCI Express (PCIe) 
devices that support the capability for message-signaled interrupts (MSI). Support for x2APIC modifies only the 
following: 

¢ the local APIC units; 

¢ the interconnects joining IOxAPIC units to the local APIC units; and 

¢ the interconnects joining MSI-capable PCI and PCIe devices to the local APIC units. 


No modifications are required to MSI-capable PCI and PCIe devices. Similarly, no modifications are required to 
IOXxAPIC units. This made possible through use of the interrupt-remapping architecture specified in the Inte/® 
Virtualization Technology for Directed I/O, Revision 1.3 for the routing of interrupts from MSI-capable devices to 
local APIC units operating in x2APIC mode. 


10.12.7 Initialization by System Software 


Routing of device interrupts to local APIC units operating in x2APIC mode requires use of the interrupt-remapping 
architecture specified in the Inte/® Virtualization Technology for Directed I/O (Revision 1.3 and/or later versions). 
Because of this, BIOS must enumerate support for and software must enable this interrupt remapping with 
Extended Interrupt Mode Enabled before it enabling x2APIC mode in the local APIC units. 


The ACPI interfaces for the x2APIC are described in Section 5.2, “ACPI System Description Tables,” of the Advanced 
Configuration and Power Interface Specification, Revision 4.0a (http://www.acpi.info/spec.htm). The default 
behavior for BIOS is to pass the control to the operating system with the local x2APICs in xAPIC mode if all APIC 
IDs reported by CPUID.OBH:EDX are less than 255, and in x2APIC mode if there are any logical processor reporting 
an APIC ID of 255 or greater. 


10.12.8 CPUID Extensions And Topology Enumeration 


For Intel 64 and IA-32 processors that support x2APIC, a value of 1 reported by CPUID.01H:ECX[21] indicates that 
the processor supports x2APIC and the extended topology enumeration leaf (CPUID.OBH). 
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The extended topology enumeration leaf can be accessed by executing CPUID with EAX = OBH. Processors that do 
not support x2APIC may support CPUID leaf OBH. Software can detect the availability of the extended topology 
enumeration leaf (OBH) by performing two steps: 


¢ Check maximum input value for basic CPUID information by executing CPUID with EAX= 0. If CPUID.OH:EAX is 
greater than or equal or 11 (OBH), then proceed to next step 


® Check CPUID.EAX=0BH, ECX=0H:EBxX is non-zero. 


If both of the above conditions are true, extended topology enumeration leaf is available. If available, the extended 
topology enumeration leaf is the preferred mechanism for enumerating topology. The presence of CPUID leaf OBH 
in a processor does not guarantee support for x2APIC. If CPUID.EAX=0BH, ECX=0H:EBX returns zero and 
maximum input value for basic CPUID information is greater than OBH, then CPUID.OBH leaf is not supported on 
that processor. 


The extended topology enumeration leaf is intended to assist software with enumerating processor topology on 
systems that requires 32-bit x2APIC IDs to address individual logical processors. Details of CPUID leaf OBH can be 
found in the reference pages of CPUID in Chapter 3 of Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 2A. 


Processor topology enumeration algorithm for processors supporting the extended topology enumeration leaf of 
CPUID and processors that do not support CPUID leaf OBH are treated in Section 8.9.4, “Algorithm for Three-Level 
Mappings of APIC_ID”. 


10.12.8.1 Consistency of APIC IDs and CPUID 


The consistency of physical x2APIC ID in MSR 802H in x2APIC mode and the 32-bit value returned in 
CPUID.OBH:EDx is facilitated by processor hardware. 


CPUID.OBH:EDxX will report the full 32 bit ID, in xAPIC and x2APIC mode. This allows BIOS to determine if a system 
has processors with IDs exceeding the 8-bit initial APIC ID limit (CPUID.O1H:EBX[31:24]). Initial APIC ID 
(CPUID.01H:EBX[31:24]) is always equal to CPUID.OBH:EDX[7:0]. 


If the values of CPUID.OBH: EDX reported by all logical processors in a system are less than 255, BIOS can transfer 
control to OS in xAPIC mode. 


If the values of CPUID.OBH:EDX reported by some logical processors in a system are greater than or equal to 255, 
BIOS must support two options to hand off to OS. 


¢ If BIOS enables logical processors with x2APIC IDs greater than 255, then it should enable x2APIC in the Boot 
Strap Processor (BSP) and all Application Processors (AP) before passing control to the OS. Applications 
requiring processor topology information must use OS provided services based on x2APIC IDs or CPUID.OBH 
leaf. 


¢ If a BIOS transfers control to OS in xAPIC mode, then the BIOS must ensure that only logical processors with 
CPUID.OBH.EDX value less than 255 are enabled. BIOS initialization on all logical processors with 
CPUID.OB.EDX values greater than or equal to 255 must (a) disable APIC and execute CLI in each logical 
processor, and (b) leave these logical processor in the lowest power state so that these processors do not 
respond to INIT IPI during OS boot. The BSP and all the enabled logical processor operate in xAPIC mode after 
BIOS passed control to OS. Application requiring processor topology information can use OS provided legacy 
services based on 8-bit initial APIC IDs or legacy topology information from CPUID.01H and CPUID O4H leaves. 
Even if the BIOS passes control in xAPIC mode, an OS can switch the processors to x2APIC mode later. BIOS 
SMM handler should always read the APIC_BASE_MSR, determine the APIC mode and use the corresponding 
access method. 


10.12.9 ICR Operation in x2APIC Mode 


In x2APIC mode, the layout of the Interrupt Command Register is shown in Figure 10-12. The lower 32 bits of ICR 
in x2APIC mode is identical to the lower half of the ICR in xAPIC mode, except the Delivery Status bit is removed 
since it is not needed in x2APIC mode. The destination ID field is expanded to 32 bits in x2APIC mode. 


To send an IPI using the ICR, software must set up the ICR to indicate the type of IPI message to be sent and the 
destination processor or processors. Self IPIs can also be sent using the SELF IPI register (see Section 10.12.11). 
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Figure 10-28. Interrupt Command Register (ICR) in x2APIC Mode 


A single MSR write to the Interrupt Command Register is required for dispatching an interrupt in x2APIC mode. 
With the removal of the Delivery Status bit, system software no longer has a reason to read the ICR. It remains 
readable only to aid in debugging; however, software should not assume the value returned by reading the ICR is 
the last written value. 


A destination ID value of FFFF_FFFFH is used for broadcast of interrupts in both logical destination and physical 
destination modes. 


10.12.10 Determining IPI Destination in x2APIC Mode 


10.12.10.1 Logical Destination Mode in x2APIC Mode 


In x2APIC mode, the Logical Destination Register (LDR) is increased to 32 bits wide. It is a read-only register to 
system software. This 32-bit value is referred to as “logical x2APIC ID”. System software accesses this register via 
the RDMSR instruction reading the MSR at address 80DH. Figure 10-29 provides the layout of the Logical Destina- 
tion Register in x2APIC mode. 
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Figure 10-29. Logical Destination Register in x2APIC Mode 


In the xAPIC mode, the Destination Format Register (DFR) through the MMIO interface determines the choice of a 
flat logical mode or a clustered logical mode. Flat logical mode is not supported in the x2APIC mode. Hence the 
Destination Format Register (DFR) is eliminated in x2APIC mode. 


The 32-bit logical x2APIC ID field of LDR is partitioned into two sub-fields: 
* Cluster ID (LDR[31:16]): is the address of the destination cluster 


¢ Logical ID (LDR[15:0]): defines a logical ID of the individual local x2APIC within the cluster specified by 
LDR[31:16]. 


This layout enables 2“16-1 clusters each with up to 16 unique logical IDs - effectively providing an addressability 
of ((2%20) - 16) processors in logical destination mode. 


It is likely that processor implementations may choose to support less than 16 bits of the cluster ID or less than 16- 
bits of the Logical ID in the Logical Destination Register. However system software should be agnostic to the 
number of bits implemented in the cluster ID and logical ID sub-fields. The x2APIC hardware initialization will 
ensure that the appropriately initialized logical x2APIC IDs are available to system software and reads of non- 
implemented bits return zero. This is a read-only register that software must read to determine the logical x2APIC 
ID of the processor. Specifically, software can apply a 16-bit mask to the lowest 16 bits of the logical x2APIC ID to 
identify the logical address of a processor within a cluster without needing to know the number of implemented bits 
in cluster ID and Logical ID sub-fields. Similarly, software can create a message destination address for cluster 
model, by bit-Oring the Logical X2APIC ID (31:0) of processors that have matching Cluster ID(31:16). 


To enable cluster ID assignment in a fashion that matches the system topology characteristics and to enable effi- 
cient routing of logical mode lowest priority device interrupts in link based platform interconnects, the LDR are 
initialized by hardware based on the value of x2APIC ID upon x2APIC state transitions. Details of this initialization 
are provided in Section 10.12.10.2. 


10.12.10.2 Deriving Logical x2APIC ID from the Local x2APIC ID 


In x2APIC mode, the 32-bit logical x2APIC ID, which can be read from LDR, is derived from the 32-bit local x2APIC 
ID. Specifically, the 16-bit logical ID sub-field is derived by shifting 1 by the lowest 4 bits of the x2APIC ID, i.e. 
Logical ID = 1 « x2APIC ID[3:0]. The remaining bits of the x2APIC ID then form the cluster ID portion of the logical 
x2APIC ID: 


Logical x2APIC ID = [(x2APIC ID[19:4] « 16) | (1 « x2APIC ID[3:0])] 


The use of the lowest 4 bits in the x2APIC ID implies that at least 16 APIC IDs are reserved for logical processors 
within a socket in multi-socket configurations. If more than 16 APIC IDS are reserved for logical processors in a 
socket/package then multiple cluster IDs can exist within the package. 


The LDR initialization occurs whenever the x2APIC mode is enabled (see Section 10.12.5). 
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10.12.11 SELF IPI Register 


SELF IPIs are used extensively by some system software. The x2APIC architecture introduces a new register inter- 
face. This new register is dedicated to the purpose of sending self-IPIs with the intent of enabling a highly opti- 
mized path for sending self-IPIs. 


Figure 10-30 provides the layout of the SELF IPI register. System software only specifies the vector associated with 
the interrupt to be sent. The semantics of sending a self-IPI via the SELF IPI register are identical to sending a self 
targeted edge triggered fixed interrupt with the specified vector. Specifically the semantics are identical to the 
following settings for an inter-processor interrupt sent via the ICR - Destination Shorthand (ICR[19:18] = 01 
(Self)), Trigger Mode (ICR[15] = 0 (Edge)), Delivery Mode (ICR[10:8] = 000 (Fixed)), Vector (ICR[7:0] = Vector). 


MSR Address: 083FH 


31 87 0 


Reserved Vector 


Figure 10-30. SELF IPI register 


The SELF IPI register is a write-only register. A RDMSR instruction with address of the SELF IPI register causes a 
general-protection exception. 


The handling and prioritization of a self-IPI sent via the SELF IPI register is architecturally identical to that for an 
IPI sent via the ICR from a legacy xAPIC unit. Specifically the state of the interrupt would be tracked via the Inter- 
rupt Request Register (IRR) and In Service Register (ISR) and Trigger Mode Register (TMR) as if it were received 
from the system bus. Also sending the IPI via the Self Interrupt Register ensures that interrupt is delivered to the 
processor core. Specifically completion of the WRMSR instruction to the SELF IPI register implies that the interrupt 
has been logged into the IRR. As expected for edge triggered interrupts, depending on the processor priority and 
readiness to accept interrupts, it is possible that interrupts sent via the SELF IPI register or via the ICR with iden- 
tical vectors can be combined. 


10.13 APIC BUS MESSAGE FORMATS 


This section describes the message formats used when transmitting messages on the serial APIC bus. The informa- 
tion described here pertains only to the Pentium and P6 family processors. 


10.13.1 Bus Message Formats 


The local and I/O APICs transmit three types of messages on the serial APIC bus: EOI message, short message, 
and non-focused lowest priority message. The purpose of each type of message and its format are described below. 


10.13.2 EOI Message 


Local APICs send 14-cycle EOI messages to the I/O APIC to indicate that a level triggered interrupt has been 
accepted by the processor. This interrupt, in turn, is a result of software writing into the EOI register of the local 
APIC. Table 10-1 shows the cycles in an EOI message. 


Table 10-1. EOI Message (14 Cycles) 


Cycle Bit1 BitO 
1 1 1 11 =€0I 
2 ArbID3 0 Arbitration ID bits 3 through O 
3 ArbID2 0 
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Table 10-1. EOI Message (14 Cycles) (Contd.) 


Cycle Bit1 BitO 
4 ArblD1 0 
5 ArbIDO 0 
6 V7 V6 Interrupt vector V7 - VO 
7 V5 V4 
8 V3 V2 
9 V1 vo 
10 C C Checksum for cycles 6 - 9 
11 0 
12 Status Cycle 0 
13 Al Al Status Cycle 1 
14 0 0 Idle 


The checksum is computed for cycles 6 through 9. It is a cumulative sum of the 2-bit (Bit1:BitO) logical data values. 
The carry out of all but the last addition is added to the sum. If any APIC computes a different checksum than the 
one appearing on the bus in cycle 10, it signals an error, driving 11 on the APIC bus during cycle 12. In this case, 
the APICs disregard the message. The sending APIC will receive an appropriate error indication (see Section 


10.5.3, “Error Handling”) and resend the message. The status cycles are defined in Table 10-4. 


10.13.2.1 Short Message 


Short messages (21-cycles) are used for sending fixed, NMI, SMI, INIT, start-up, ExtINT and lowest-priority-with- 
focus interrupts. Table 10-2 shows the cycles in a short message. 


Table 10-2. Short Message (21 Cycles) 


Cycle Bit1 BitO 
1 0 1 0 1 =normal 
2 ArbID3 0 Arbitration ID bits 3 through O 
3 ArbID2 0 
4 ArbID1 0 
5 ArbIDO 0 
6 DM M2 DM = Destination Mode 
7 M1 MO M2-MO = Delivery mode 
8 L ™ L = Level, TM = Trigger Mode 
9 V7 V6 V7-VO = Interrupt Vector 
10 V5 V4 
11 V3 V2 
12 V1 vo 
13 D7 D6 D7-DO = Destination 
14 D5 D4 
15 D3 D2 
16 D1 DO 
17 C C Checksum for cycles 6-16 
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Table 10-2. Short Message (21 Cycles) (Contd.) 


Cycle Bit1 BitO 
18 0 0 
19 A A Status cycle 0 
20 Al Al Status cycle 1 
21 0 0 Idle 


If the physical delivery mode is being used, then cycles 15 and 16 represent the APIC ID and cycles 13 and 14 are 
considered don't care by the receiver. If the logical delivery mode is being used, then cycles 13 through 16 are the 
8-bit logical destination field. 


For shorthands of “all-incl-self” and “all-excl-self,” the physical delivery mode and an arbitration priority of 15 
(DO:D3 = 1111) are used. The agent sending the message is the only one required to distinguish between the two 
cases. It does so using internal information. 


When using lowest priority delivery with an existing focus processor, the focus processor identifies itself by driving 
10 during cycle 19 and accepts the interrupt. This is an indication to other APICs to terminate arbitration. If the 
focus processor has not been found, the short message is extended on-the-fly to the non-focused lowest-priority 
message. Note that except for the EOI message, messages generating a checksum or an acceptance error (see 
Section 10.5.3, “Error Handling”) terminate after cycle 21. 


10.13.2.2 Non-focused Lowest Priority Message 


These 34-cycle messages (see Table 10-3) are used in the lowest priority delivery mode when a focus processor is 
not present. Cycles 1 through 20 are same as for the short message. If during the status cycle (cycle 19) the state 
of the (A:A) flags is 10B, a focus processor has been identified, and the short message format is used (see Table 
10-2). If the (A:A) flags are set to 00B, lowest priority arbitration is started and the 34-cycles of the non-focused 
lowest priority message are competed. For other combinations of status flags, refer to Section 10.13.2.3, “APIC 
Bus Status Cycles.” 


Table 10-3. Non-Focused Lowest Priority Message (34 Cycles) 


Cycle BitO Bit1 
1 0 1 0 1 = normal 
2 ArbID3 0 Arbitration ID bits 3 through O 
3 ArbID2 0 
4 ArbID1 0 
5 ArbIDO 0 
6 DM M2 DM = Destination mode 
7 M1 MO M2-MO = Delivery mode 
8 L ™ L = Level, TM = Trigger Mode 
9 V7 V6 V7-VO = Interrupt Vector 
10 V5 V4 
11 V3 V2 
12 V1 vo 
13 D7 D6 D7-DO = Destination 
14 D5 D4 
15 D3 D2 
16 D1 DO 
17 C C Checksum for cycles 6-16 
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Table 10-3. Non-Focused Lowest Priority Message (34 Cycles) (Contd.) 


Cycle BitO Bit1 
18 0 0 
19 A A Status cycle 0 
20 Al Al Status cycle 1 
21 P7 0 P7 - PO = Inverted Processor Priority 
22 P6 0 
23 P5 0 
24 P4 0 
25 P3 0 
26 P2 0 
27 P1 0 
28 PO 0 
29 ArbID3 0 Arbitration ID 3 -0 
30 ArbID2 0 
31 ArbiD1 0 
32 ArbIDO 0 
33 A2 A2 Status Cycle 
34 0 0 Idle 


Cycles 21 through 28 are used to arbitrate for the lowest priority processor. The processors participating in the 
arbitration drive their inverted processor priority on the bus. Only the local APICs having free interrupt slots partic- 
ipate in the lowest priority arbitration. If no such APIC exists, the message will be rejected, requiring it to be tried 
at a later time. 


Cycles 29 through 32 are also used for arbitration in case two or more processors have the same lowest priority. In 
the lowest priority delivery mode, all combinations of errors in cycle 33 (A2 A2) will set the “accept error” bit in the 
error status register (see Figure 10-9). Arbitration priority update is performed in cycle 20, and is not affected by 
errors detected in cycle 33. Only the local APIC that wins in the lowest priority arbitration, drives cycle 33. An error 
in cycle 33 will force the sender to resend the message. 


10.13.2.3 APIC Bus Status Cycles 


Certain cycles within an APIC bus message are status cycles. During these cycles the status flags (A:A) and 
(A1:A1) are examined. Table 10-4 shows how these status flags are interpreted, depending on the current delivery 
mode and existence of a focus processor. 
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Table 10-4. APIC Bus Status Cycles Interpretation 


Delivery A Status Al Status A2 Status Update ArbID | Message Retry 
Mode and Cycle# Length 
EOI 00: CS_OK 10: Accept XX: Yes, 13 14 Cycle No 
00: CS_OK 11: Retry XX: Yes, 13 14 Cycle Yes 
00: CS_OK OX: Accept Error XX: No 14 Cycle Yes 
11: CS_Error XX: XX: No 14 Cycle Yes 
10: Error XX: XX: No 14 Cycle Yes 
01: Error XX: XX: No 14 Cycle Yes 
Fixed 00: CS_OK 10: Accept XX: Yes, 20 21 Cycle No 
00: CS_OK 11: Retry XX: Yes, 20 21 Cycle Yes 
00: CS_OK OX: Accept Error XX: No 21 Cycle Yes 
11: CS_Error XX: XX: No 21 Cycle Yes 
10: Error XX: XX: No 21 Cycle Yes 
01: Error XX: XX: No 21 Cycle Yes 
NMI, SMI, INIT, 00: CS_OK 10: Accept XX: Yes, 20 21 Cycle No 
Sats 00: CS_OK 11: Retry XX: Yes, 20 21 Cycle Yes 
00: CS_OK OX: Accept Error XX: No 21 Cycle Yes 
11: CS_Error XX: XX: No 21 Cycle Yes 
10: Error XX: XX: No 21 Cycle Yes 
01: Error XX: XX: No 21 Cycle Yes 
Lowest 00: CS_OK, NoFocus | 11: Do Lowest 10: Accept Yes, 20 34 Cycle No 
00: CS_OK, NoFocus | 11: Do Lowest 11: Error Yes, 20 34 Cycle Yes 
00: CS_OK, NoFocus | 11: Do Lowest OX: Error Yes, 20 34 Cycle Yes 
00: CS_OK, NoFocus | 10: End and Retry XX: Yes, 20 34 Cycle Yes 
00: CS_OK, NoFocus | OX: Error XX: No 34 Cycle Yes 
10: CS_OK, Focus XX: XX: Yes, 20 34 Cycle No 
11: CS_Error XX: XX: No 21 Cycle Yes 
01: Error XX: XX: No 21 Cycle Yes 
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CHAPTER 11 
MEMORY CACHE CONTROL 


This chapter describes the memory cache and cache control mechanisms, the TLBs, and the store buffer in Intel 64 
and IA-32 processors. It also describes the memory type range registers (MTRRs) introduced in the P6 family 
processors and how they are used to control caching of physical memory locations. 


11.1. INTERNAL CACHES, TLBS, AND BUFFERS 


The Intel 64 and IA-32 architectures support cache, translation look aside buffers (TLBs), and a store buffer for 
temporary on-chip (and external) storage of instructions and data. (Figure 11-1 shows the arrangement of caches, 
TLBs, and the store buffer for the Pentium 4 and Intel Xeon processors.) Table 11-1 shows the characteristics of 
these caches and buffers for the Pentium 4, Intel Xeon, P6 family, and Pentium processors. The sizes and char- 
acteristics of these units are machine specific and may change in future versions of the processor. The 
CPUID instruction returns the sizes and characteristics of the caches and buffers for the processor on which the 
instruction is executed. See “CPUID—CPU Identification” in Chapter 3, “Instruction Set Reference, A-L,” of the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A. 
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Figure 11-1. Cache Structure of the Pentium 4 and Intel Xeon Processors 
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Figure 11-2. Cache Structure of the Intel Core i7 Processors 


Figure 11-2 shows the cache arrangement of Intel Core i7 processor. 


Table 11-1. Characteristics of the Caches, TLBs, Store Buffer, and 
Write Combining Buffer in Intel 64 and IA-32 Processors 


Cache or Buffer 


Characteristics 


Trace Cache! 


« Pentium 4 and Intel Xeon processors (Based on Intel NetBurst® microarchitecture): 12 Kuwops, 8-way set 
associative. 

« Intel Core i7, Intel Core 2 Duo, Intel® Atom™, Intel Core Duo, Intel Core Solo, Pentium M processor: not 
implemented. 

« P6 family and Pentium processors: not implemented. 


L1 Instruction Cache 


* Pentium 4 and Intel Xeon processors (Based on Intel NetBurst microarchitecture): not implemented. 

« Intel Core i7 processor: 32-KByte, 4-way set associative. 

« Intel Core 2 Duo, Intel Atom, Intel Core Duo, Intel Core Solo, Pentium M processor: 32-KByte, 8-way set 
associative. 

= P6 family and Pentium processors: 8- or 16-KByte, 4-way set associative, 32-byte cache line size; 2-way set 
associative for earlier Pentium processors. 


L1 Data Cache 


* Pentium 4 and Intel Xeon processors (Based on Intel NetBurst microarchitecture): 8-KByte, 4-way set 
associative, 64-byte cache line size. 

* Pentium 4 and Intel Xeon processors (Based on Intel NetBurst microarchitecture): 16-KByte, 8-way set 
associative, 64-byte cache line size. 

« Intel Atom processors: 24-KByte, 6-way set associative, 64-byte cache line size. 

« Intel Core i7, Intel Core 2 Duo, Intel Core Duo, Intel Core Solo, Pentium M and Intel Xeon processors: 32- 
KByte, 8-way set associative, 64-byte cache line size. 

= P6 family processors: 16-KByte, 4-way set associative, 32-byte cache line size; 8-KBytes, 2-way set 
associative for earlier P6 family processors. 

« Pentium processors: 16-KByte, 4-way set associative, 32-byte cache line size; 8-KByte, 2-way set 
associative for earlier Pentium processors. 
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Table 11-1. Characteristics of the Caches, TLBs, Store Buffer, and 
Write Combining Buffer in Intel 64 and IA-32 Processors (Contd.) 


Cache or Buffer 


Characteristics 


L2 Unified Cache 


Intel Core 2 Duo and Intel Xeon processors: up to 4-MByte (or 4MBx2 in quadcore processors), 16-way set 
associative, 64-byte cache line size. 

Intel Core 2 Duo and Intel Xeon processors: up to 6-MByte (or 6MBx2 in quadcore processors), 24-way set 
associative, 64-byte cache line size. 

Intel Core i7, i5, i3 processors: 256KBbyte, 8-way set associative, 64-byte cache line size. 

Intel Atom processors: 512-KByte, 8-way set associative, 64-byte cache line size. 

Intel Core Duo, Intel Core Solo processors: 2-MByte, 8-way set associative, 64-byte cache line size 

Pentium 4 and Intel Xeon processors: 256, 512, 1024, or 2048-KByte, 8-way set associative, 64-byte cache 
line size, 128-byte sector size. 

Pentium M processor: 1 or 2-MByte, 8-way set associative, 64-byte cache line size. 

P6 family processors: 128-KByte, 256-KByte, 512-KByte, 1-MByte, or 2-MByte, 4-way set associative, 
32-byte cache line size. 

Pentium processor (external optional): System specific, typically 256- or 512-KByte, 4-way set associative, 
32-byte cache line size. 


L3 Unified Cache 


Intel Xeon processors: 512-KByte, 1-MByte, 2-MByte, or 4-MByte, 8-way set associative, 64-byte cache line 
size, 128-byte sector size. 

Intel Core i7 processor, Intel Xeon processor 5500: Up to 8MByte, 16-way set associative, 64-byte cache 
line size. 

Intel Xeon processor 5600: Up to 12MByte, 64-byte cache line size. 

Intel Xeon processor 7500: Up to 24MByte, 64-byte cache line size. 


Instruction TLB 
(4-KByte Pages) 


Pentium 4 and Intel Xeon processors (Based on Intel NetBurst microarchitecture): 128 entries, 4-way set 
associative. 

Intel Atom processors: 32-entries, fully associative. 

Intel Core i7, i5, i3 processors: 64-entries per thread (128-entries per core), 4-way set associative. 

Intel Core 2 Duo, Intel Core Duo, Intel Core Solo processors, Pentium M processor: 128 entries, 4-way set 
associative. 

P6 family processors: 32 entries, 4-way set associative. 

Pentium processor: 32 entries, 4-way set associative; fully set associative for Pentium processors with MMX 
technology. 


Data TLB (4-KByte 
Pages) 


« Intel Atom processors: 16-entry-per-thread micro-TLB, fully associative; 64-entry DTLB, 4-way set 


Intel Core i7, i5, i3 processors, DTLBO: 64-entries, 4-way set associative. 
Intel Core 2 Duo processors: DTLBO, 16 entries, DTLB1, 256 entries, 4 ways. 


associative; 16-entry PDE cache, fully associative. 

Pentium 4 and Intel Xeon processors (Based on Intel NetBurst microarchitecture): 64 entry, fully set 
associative, shared with large page DTLB. 

Intel Core Duo, Intel Core Solo processors, Pentium M processor: 128 entries, 4-way set associative. 
Pentium and P6 family processors: 64 entries, 4-way set associative; fully set, associative for Pentium 
processors with MMX technology. 


Instruction TLB 
(Large Pages) 


Intel Core i7, i5, i3 processors: 7-entries per thread, fully associative. 

Intel Core 2 Duo processors: 4 entries, 4 ways. 

Pentium 4 and Intel Xeon processors: large pages are fragmented. 

Intel Core Duo, Intel Core Solo, Pentium M processor: 2 entries, fully associative. 
P6 family processors: 2 entries, fully associative. 

Pentium processor: Uses same TLB as used for 4-KByte pages. 


Data TLB (Large 
Pages) 


Intel Core i7, i5, i3 processors, DTLBO: 32-entries, 4-way set associative. 

Intel Core 2 Duo processors: DTLBO, 16 entries, DTLB1, 32 entries, 4 ways. 

Intel Atom processors: 8 entries, 4-way set associative. 

Pentium 4 and Intel Xeon processors: 64 entries, fully set associative; shared with small page data TLBs. 
Intel Core Duo, Intel Core Solo, Pentium M processor: 8 entries, fully associative. 

P6 family processors: 8 entries, 4-way set associative. 

Pentium processor: 8 entries, 4-way set associative; uses same TLB as used for 4-KByte pages in Pentium 
processors with MMX technology. 


Second-level Unified 
TLB (4-KByte 
Pages) 


Intel Core i7, i5, i3 processor, STLB: 512-entries, 4-way set associative. 
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Table 11-1. Characteristics of the Caches, TLBs, Store Buffer, and 
Write Combining Buffer in Intel 64 and IA-32 Processors (Contd.) 


Cache or Buffer Characteristics 


Store Buffer 


« Intel Core i7, i5, i3 processors: 32entries. 

« Intel Core 2 Duo processors: 20 entries. 

« Intel Atom processors: 8 entries, used for both WC and store buffers. 

« Pentium 4 and Intel Xeon processors: 24 entries. 

« Pentium M processor: 16 entries. 

= P6 family processors: 12 entries. 

« Pentium processor: 2 buffers, 1 entry each (Pentium processors with MMX technology have 4 buffers for 4 


entries). 
Write Combining « Intel Core 2 Duo processors: 8 entries. 
(WC) Buffer « Intel Atom processors: 8 entries, used for both WC and store buffers. 
« Pentium 4 and Intel Xeon processors: 6 or 8 entries. 
« Intel Core Duo, Intel Core Solo, Pentium M processors: 6 entries. 
= P6 family processors: 4 entries. 
NOTES: 


1 Introduced to the IA-32 architecture in the Pentium 4 and Intel Xeon processors. 


Intel 64 and IA-32 processors may implement four types of caches: the trace cache, the level 1 (L1) cache, the 
level 2 (L2) cache, and the level 3 (L3) cache. See Figure 11-1. Cache availability is described below: 


Intel Core i7, i5, i3 processor Family and Intel Xeon processor Family based on Intel® microarchi- 
tecture code name Nehalem and Intel® microarchitecture code name Westmere — The L1 cache is 
divided into two sections: one section is dedicated to caching instructions (pre-decoded instructions) and the 
other caches data. The L2 cache is a unified data and instruction cache. Each processor core has its own L1 and 
L2. The L3 cache is an inclusive, unified data and instruction cache, shared by all processor cores inside a 
physical package. No trace cache is implemented. 


Intel® Core™ 2 processor family and Intel® Xeon® processor family based on Intel® Core™ micro- 
architecture — The L1 cache is divided into two sections: one section is dedicated to caching instructions (pre- 
decoded instructions) and the other caches data. The L2 cache is a unified data and instruction cache located 
on the processor chip; it is shared between two processor cores in a dual-core processor implementation. 
Quad-core processors have two L2, each shared by two processor cores. No trace cache is implemented. 


Intel® Atom™ processor — The L1 cache is divided into two sections: one section is dedicated to caching 
instructions (pre-decoded instructions) and the other caches data. The L2 cache is a unified data and 
instruction cache is located on the processor chip. No trace cache is implemented. 


Intel® Core™ Solo and Intel® Core™ Duo processors — The L1 cache is divided into two sections: one 
section is dedicated to caching instructions (pre-decoded instructions) and the other caches data. The L2 cache 
is a unified data and instruction cache located on the processor chip. It is shared between two processor cores 
in a dual-core processor implementation. No trace cache is implemented. 


Pentium® 4 and Intel® Xeon® processors Based on Intel NetBurst® microarchitecture — The trace 

cache caches decoded instructions (uops) from the instruction decoder and the Li cache contains data. The L2 
and L3 caches are unified data and instruction caches located on the processor chip. Dualcore processors have 
two L2, one in each processor core. Note that the L3 cache is only implemented on some Intel Xeon processors. 


P6 family processors — The L1 cache is divided into two sections: one dedicated to caching instructions (pre- 
decoded instructions) and the other to caching data. The L2 cache is a unified data and instruction cache 
located on the processor chip. P6 family processors do not implement a trace cache. 


Pentium® processors — The L1 cache has the same structure as on P6 family processors. There is no trace 
cache. The L2 cache is a unified data and instruction cache external to the processor chip on earlier Pentium 
processors and implemented on the processor chip in later Pentium processors. For Pentium processors where 
the L2 cache is external to the processor, access to the cache is through the system bus. 


For Intel Core i7 processors and processors based on Intel Core, Intel Atom, and Intel NetBurst microarchitectures, 
Intel Core Duo, Intel Core Solo and Pentium M processors, the cache lines for the L1 and L2 caches (and L3 caches 
if supported) are 64 bytes wide. The processor always reads a cache line from system memory beginning on a 64- 
byte boundary. (A 64-byte aligned cache line begins at an address with its 6 least-significant bits clear.) A cache 
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line can be filled from memory with a 8-transfer burst transaction. The caches do not support partially-filled cache 
lines, so caching even a single doubleword requires caching an entire line. 


The Li and L2 cache lines in the P6 family and Pentium processors are 32 bytes wide, with cache line reads from 
system memory beginning on a 32-byte boundary (5 least-significant bits of a memory address clear.) A cache line 
can be filled from memory with a 4-transfer burst transaction. Partially-filled cache lines are not supported. 


The trace cache in processors based on Intel NetBurst microarchitecture is available in all execution modes: 
protected mode, system management mode (SMM), and real-address mode. The Li,L2, and L3 caches are also 
available in all execution modes; however, use of them must be handled carefully in SMM (see Section 30.4.2, 
“SMRAM Caching”). 


The TLBs store the most recently used page-directory and page-table entries. They speed up memory accesses 
when paging is enabled by reducing the number of memory accesses that are required to read the page tables 
stored in system memory. The TLBs are divided into four groups: instruction TLBs for 4-KByte pages, data TLBs for 
4-KByte pages; instruction TLBs for large pages (2-MByte, 4-MByte or 1-GByte pages), and data TLBs for large 
pages. The TLBs are normally active only in protected mode with paging enabled. When paging is disabled or the 
processor is in real-address mode, the TLBs maintain their contents until explicitly or implicitly flushed (see Section 
11.9, “Invalidating the Translation Lookaside Buffers (TLBs)”). 


Processors based on Intel Core microarchitectures implement one level of instruction TLB and two levels of data 
TLB. Intel Core i7 processor provides a second-level unified TLB. 


The store buffer is associated with the processors instruction execution units. It allows writes to system memory 
and/or the internal caches to be saved and in some cases combined to optimize the processor’s bus accesses. The 
store buffer is always enabled in all execution modes. 


The processor's caches are for the most part transparent to software. When enabled, instructions and data flow 
through these caches without the need for explicit software control. However, knowledge of the behavior of these 
caches may be useful in optimizing software performance. For example, knowledge of cache dimensions and 
replacement algorithms gives an indication of how large of a data structure can be operated on at once without 
causing cache thrashing. 


In multiprocessor systems, maintenance of cache consistency may, in rare circumstances, require intervention by 
system software. For these rare cases, the processor provides privileged cache control instructions for use in 
flushing caches and forcing memory ordering. 


There are several instructions that software can use to improve the performance of the Li, L2, and L3 caches, 
including the PREFETCHh, CLFLUSH, and CLFLUSHOPT instructions and the non-temporal move instructions 
(MOVNTI, MOVNTQ, MOVNTDQ, MOVNTPS, and MOVNTPD). The use of these instructions are discussed in Section 
11.5.5, “Cache Management Instructions.” 


11.2 CACHING TERMINOLOGY 


IA-32 processors (beginning with the Pentium processor) and Intel 64 processors use the MESI (modified, exclu- 
sive, shared, invalid) cache protocol to maintain consistency with internal caches and caches in other processors 
(see Section 11.4, “Cache Control Protocol”). 


When the processor recognizes that an operand being read from memory is cacheable, the processor reads an 
entire cache line into the appropriate cache (L1, L2, L3, or all). This operation is called a cache line fill. If the 
memory location containing that operand is still cached the next time the processor attempts to access the 
operand, the processor can read the operand from the cache instead of going back to memory. This operation is 
called a cache hit. 


When the processor attempts to write an operand to a cacheable area of memory, it first checks if a cache line for 
that memory location exists in the cache. If a valid cache line does exist, the processor (depending on the write 
policy currently in force) can write the operand into the cache instead of writing it out to system memory. This 
operation is called a write hit. If a write misses the cache (that is, a valid cache line is not present for area of 
memory being written to), the processor performs a cache line fill, write allocation. Then it writes the operand into 
the cache line and (depending on the write policy currently in force) can also write it out to memory. If the operand 
is to be written out to memory, it is written first into the store buffer, and then written from the store buffer to 
memory when the system bus is available. (Note that for the Pentium processor, write misses do not result in a 
cache line fill; they always result in a write to memory. For this processor, only read misses result in cache line fills.) 
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When operating in an MP system, IA-32 processors (beginning with the Intel486 processor) and Intel 64 processors 
have the ability to snoop other processor’s accesses to system memory and to their internal caches. They use this 
snooping ability to keep their internal caches consistent both with system memory and with the caches in other 
processors on the bus. For example, in the Pentium and P6 family processors, if through snooping one processor 
detects that another processor intends to write to a memory location that it currently has cached in shared state, 
the snooping processor will invalidate its cache line forcing it to perform a cache line fill the next time it accesses 
the same memory location. 


Beginning with the P6 family processors, if a processor detects (through snooping) that another processor is trying 
to access a memory location that it has modified in its cache, but has not yet written back to system memory, the 
snooping processor will signal the other processor (by means of the HITM# signal) that the cache line is held in 
modified state and will perform an implicit write-back of the modified data. The implicit write-back is transferred 
directly to the initial requesting processor and snooped by the memory controller to assure that system memory 
has been updated. Here, the processor with the valid data may pass the data to the other processors without actu- 
ally writing it to system memory; however, it is the responsibility of the memory controller to snoop this operation 
and update memory. 


11.3. METHODS OF CACHING AVAILABLE 


The processor allows any area of system memory to be cached in the L1, L2, and L3 caches. In individual pages or 
regions of system memory, it allows the type of caching (also called memory type) to be specified (see Section 
11.5). Memory types currently defined for the Intel 64 and IA-32 architectures are (see Table 11-2): 


¢ Strong Uncacheable (UC) —System memory locations are not cached. All reads and writes appear on the 
system bus and are executed in program order without reordering. No speculative memory accesses, page- 
table walks, or prefetches of speculated branch targets are made. This type of cache-control is useful for 
memory-mapped I/O devices. When used with normal RAM, it greatly reduces processor performance. 


NOTE 


The behavior of x87 and SIMD instructions referencing memory is implementation dependent. In 
some implementations, accesses to UC memory may occur more than once. To ensure predictable 
behavior, use loads and stores of general purpose registers to access UC memory that may have 
read or write side effects. 


Table 11-2. Memory Types and Their Properties 


Memory Type and Cacheable | Writeback | Allows Memory Ordering Model 
Mnemonic Cacheable | Speculative 
Reads 
Strong Uncacheable | No No No Strong Ordering 
(UC) 
Uncacheable (UC-) No No No Strong Ordering. Can only be selected through the PAT. Can be 
overridden by WC in MTRRs. 
Write Combining (WC) | No No Yes Weak Ordering. Available by programming MTRRs or by selecting it 
through the PAT. 
Write Through (WT) | Yes No Yes Speculative Processor Ordering. 
Write Back (WB) Yes Yes Yes Speculative Processor Ordering. 
Write Protected (WP) | Yes for No Yes Speculative Processor Ordering. Available by programming MTRRs. 
reads; no for 
writes 


¢ Uncacheable (UC-) — Has same characteristics as the strong uncacheable (UC) memory type, except that 
this memory type can be overridden by programming the MTRRs for the WC memory type. This memory type 
is available in processor families starting from the Pentium III processors and can only be selected through the 
PAT. 
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¢ Write Combining (WC) — System memory locations are not cached (as with uncacheable memory) and 

coherency is not enforced by the processor’s bus coherency protocol. Speculative reads are allowed. Writes 
may be delayed and combined in the write combining buffer (WC buffer) to reduce memory accesses. If the WC 
buffer is partially filled, the writes may be delayed until the next occurrence of a serializing event; such as an 
SFENCE or MFENCE instruction, CPUID or other serializing instruction, a read or write to uncached memory, an 
interrupt occurrence, or an execution of a LOCK instruction (including one with an XACQUIRE or XRELEASE 
prefix). In addition, an execution of the XEND instruction (to end a transactional region) evicts any writes that 
were buffered before the corresponding execution of the XBEGIN instruction (to begin the transactional region) 
before evicting any writes that were performed inside the transactional region. 


This type of cache-control is appropriate for video frame buffers, where the order of writes is unimportant as 
long as the writes update memory so they can be seen on the graphics display. See Section 11.3.1, “Buffering 
of Write Combining Memory Locations,” for more information about caching the WC memory type. This memory 
type is available in the Pentium Pro and Pentium II processors by programming the MTRRs; or in processor 
families starting from the Pentium III processors by programming the MTRRs or by selecting it through the PAT. 


¢ Write-through (WT) — Writes and reads to and from system memory are cached. Reads come from cache 
lines on cache hits; read misses cause cache fills. Speculative reads are allowed. All writes are written to a 
cache line (when possible) and through to system memory. When writing through to memory, invalid cache 
lines are never filled, and valid cache lines are either filled or invalidated. Write combining is allowed. This type 
of cache-control is appropriate for frame buffers or when there are devices on the system bus that access 
system memory, but do not perform snooping of memory accesses. It enforces coherency between caches in 
the processors and system memory. 


¢ Write-back (WB) — Writes and reads to and from system memory are cached. Reads come from cache lines 
on cache hits; read misses cause cache fills. Speculative reads are allowed. Write misses cause cache line fills 
(in processor families starting with the P6 family processors), and writes are performed entirely in the cache, 
when possible. Write combining is allowed. The write-back memory type reduces bus traffic by eliminating 
many unnecessary writes to system memory. Writes to a cache line are not immediately forwarded to system 
memory; instead, they are accumulated in the cache. The modified cache lines are written to system memory 
later, when a write-back operation is performed. Write-back operations are triggered when cache lines need to 
be deallocated, such as when new cache lines are being allocated in a cache that is already full. They also are 
triggered by the mechanisms used to maintain cache consistency. This type of cache-control provides the best 
performance, but it requires that all devices that access system memory on the system bus be able to snoop 
memory accesses to ensure system memory and cache coherency. 


¢ Write protected (WP) — Reads come from cache lines when possible, and read misses cause cache fills. 
Writes are propagated to the system bus and cause corresponding cache lines on all processors on the bus to 
be invalidated. Speculative reads are allowed. This memory type is available in processor families starting from 
the P6 family processors by programming the MTRRs (see Table 11-6). 


Table 11-3 shows which of these caching methods are available in the Pentium, P6 Family, Pentium 4, and Intel 
Xeon processors. 


Table 11-3. Methods of Caching Available in Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium M, Pentium 4, 
Intel Xeon, P6 Family, and Pentium Processors 


Memory Type Intel Core 2 Duo, Intel Atom, Intel Core Duo, P6 Family Pentium 
Pentium M, Pentium 4 and Intel Xeon Processors | Processors Processor 

Strong Uncacheable (UC) Yes Yes Yes 
Uncacheable (UC-) Yes Yes* No 

Write Combining (WC) Yes Yes No 

Write Through (WT) Yes Yes Yes 

Write Back (WB) Yes Yes Yes 

Write Protected (WP) Yes Yes No 

NOTE: 


* Introduced in the Pentium Ill processor; not available in the Pentium Pro or Pentium II processors 
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11.3.1 Buffering of Write Combining Memory Locations 


Writes to the WC memory type are not cached in the typical sense of the word cached. They are retained in an 
internal write combining buffer (WC buffer) that is separate from the internal L1, L2, and L3 caches and the store 
buffer. The WC buffer is not snooped and thus does not provide data coherency. Buffering of writes to WC memory 
is done to allow software a small window of time to supply more modified data to the WC buffer while remaining as 
non-intrusive to software as possible. The buffering of writes to WC memory also causes data to be collapsed; that 
is, multiple writes to the same memory location will leave the last data written in the location and the other writes 
will be lost. 


The size and structure of the WC buffer is not architecturally defined. For the Intel Core 2 Duo, Intel Atom, Intel 
Core Duo, Pentium M, Pentium 4 and Intel Xeon processors; the WC buffer is made up of several 64-byte WC 
buffers. For the P6 family processors, the WC buffer is made up of several 32-byte WC buffers. 


When software begins writing to WC memory, the processor begins filling the WC buffers one at a time. When one 
or more WC buffers has been filled, the processor has the option of evicting the buffers to system memory. The 
protocol for evicting the WC buffers is implementation dependent and should not be relied on by software for 
system memory coherency. When using the WC memory type, software must be sensitive to the fact that the 
writing of data to system memory is being delayed and must deliberately empty the WC buffers when system 
memory coherency is required. 


Once the processor has started to evict data from the WC buffer into system memory, it will make a bus-transaction 
style decision based on how much of the buffer contains valid data. If the buffer is full (for example, all bytes are 
valid), the processor will execute a burst-write transaction on the bus. This results in all 32 bytes (P6 family proces- 
sors) or 64 bytes (Pentium 4 and more recent processor) being transmitted on the data bus in a single burst trans- 
action. If one or more of the WC buffer’s bytes are invalid (for example, have not been written by software), the 
processor will transmit the data to memory using “partial write” transactions (one chunk at a time, where a “chunk” 
is 8 bytes). 


This will result ina maximum of 4 partial write transactions (for P6 family processors) or 8 partial write transactions 
(for the Pentium 4 and more recent processors) for one WC buffer of data sent to memory. 


The WC memory type is weakly ordered by definition. Once the eviction of a WC buffer has started, the data is 
subject to the weak ordering semantics of its definition. Ordering is not maintained between the successive alloca- 
tion/deallocation of WC buffers (for example, writes to WC buffer 1 followed by writes to WC buffer 2 may appear 
as buffer 2 followed by buffer 1 on the system bus). When a WC buffer is evicted to memory as partial writes there 
is no guaranteed ordering between successive partial writes (for example, a partial write for chunk 2 may appear 
on the bus before the partial write for chunk 1 or vice versa). 


The only elements of WC propagation to the system bus that are guaranteed are those provided by transaction 
atomicity. For example, with a P6 family processor, a completely full WC buffer will always be propagated as a 
single 32-bit burst transaction using any chunk order. In a WC buffer eviction where data will be evicted as partials, 
all data contained in the same chunk (0 mod 8 aligned) will be propagated simultaneously. Likewise, for more 
recent processors starting with those based on Intel NetBurst microarchitectures, a full WC buffer will always be 
propagated as a single burst transactions, using any chunk order within a transaction. For partial buffer propaga- 
tions, all data contained in the same chunk will be propagated simultaneously. 


11.3.2 Choosing a Memory Type 


The simplest system memory model does not use memory-mapped I/O with read or write side effects, does not 
include a frame buffer, and uses the write-back memory type for all memory. An I/O agent can perform direct 
memory access (DMA) to write-back memory and the cache protocol maintains cache coherency. 


A system can use strong uncacheable memory for other memory-mapped I/O, and should always use strong unca- 
cheable memory for memory-mapped I/O with read side effects. 


Dual-ported memory can be considered a write side effect, making relatively prompt writes desirable, because 
those writes cannot be observed at the other port until they reach the memory agent. A system can use strong 
uncacheable, uncacheable, write-through, or write-combining memory for frame buffers or dual-ported memory 
that contains pixel values displayed on a screen. Frame buffer memory is typically large (a few megabytes) and is 
usually written more than it is read by the processor. Using strong uncacheable memory for a frame buffer gener- 
ates very large amounts of bus traffic, because operations on the entire buffer are implemented using partial writes 
rather than line writes. Using write-through memory for a frame buffer can displace almost all other useful cached 
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lines in the processor's L2 and L3 caches and L1 data cache. Therefore, systems should use write-combining 
memory for frame buffers whenever possible. 


Software can use page-level cache control, to assign appropriate effective memory types when software will not 
access data structures in ways that benefit from write-back caching. For example, software may read a large data 
structure once and not access the structure again until the structure is rewritten by another agent. Such a large 
data structure should be marked as uncacheable, or reading it will evict cached lines that the processor will be 
referencing again. 


A similar example would be a write-only data structure that is written to (to export the data to another agent), but 
never read by software. Such a structure can be marked as uncacheable, because software never reads the values 
that it writes (though as uncacheable memory, it will be written using partial writes, while as write-back memory, 
it will be written using line writes, which may not occur until the other agent reads the structure and triggers 
implicit write-backs). 


On the Pentium Ill, Pentium 4, and more recent processors, new instructions are provided that give software 
greater control over the caching, prefetching, and the write-back characteristics of data. These instructions allow 
software to use weakly ordered or processor ordered memory types to improve processor performance, but when 
necessary to force strong ordering on memory reads and/or writes. They also allow software greater control over 
the caching of data. For a description of these instructions and there intended use, see Section 11.5.5, “Cache 
Management Instructions.” 


11.3.3. Code Fetches in Uncacheable Memory 


Programs may execute code from uncacheable (UC) memory, but the implications are different from accessing 
data in UC memory. When doing code fetches, the processor never transitions from cacheable code to UC code 
speculatively. It also never speculatively fetches branch targets that result in UC code. 


The processor may fetch the same UC cache line multiple times in order to decode an instruction once. It may 
decode consecutive UC instructions in a cacheline without fetching between each instruction. It may also fetch 
additional cachelines from the same or a consecutive 4-KByte page in order to decode one non-speculative UC 
instruction (this can be true even when the instruction is contained fully in one line). 


Because of the above and because cacheline sizes may change in future processors, software should avoid placing 
memory-mapped I/O with read side effects in the same page or in a subsequent page used to execute UC code. 


11.4 CACHE CONTROL PROTOCOL 


The following section describes the cache control protocol currently defined for the Intel 64 and IA-32 architec- 
tures. 


In the Li data cache and in the L2/L3 unified caches, the MESI (modified, exclusive, shared, invalid) cache protocol 
maintains consistency with caches of other processors. The L1 data cache and the L2/L3 unified caches have two 
MESI status flags per cache line. Each line can be marked as being in one of the states defined in Table 11-4. In 
general, the operation of the MESI protocol is transparent to programs. 


Table 11-4. MESI Cache Line States 


Cache Line State M (Modified) E (Exclusive) S (Shared) | (Invalid) 
This cache line is valid? Yes Yes Yes No 
The memory copy is... Out of date Valid Valid - 
Copies exist in caches of other | No No Maybe Maybe 
processors? 
A write to this line ... Does not go to the Does not go to the Causes the processor to Goes directly to the 
system bus. system bus. gain exclusive ownership | system bus. 
of the line. 
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The L1 instruction cache in P6 family processors implements only the “SI” part of the MESI protocol, because the 
instruction cache is not writable. The instruction cache monitors changes in the data cache to maintain consistency 
between the caches when instructions are modified. See Section 11.6, “Self-Modifying Code,” for more information 
on the implications of caching instructions. 


11.5 CACHE CONTROL 


The Intel 64 and IA-32 architectures provide a variety of mechanisms for controlling the caching of data and 
instructions and for controlling the ordering of reads and writes between the processor, the caches, and memory. 
These mechanisms can be divided into two groups: 


¢ Cache control registers and bits — The Intel 64 and IA-32 architectures define several dedicated registers 
and various bits within control registers and page- and directory-table entries that control the caching system 
memory locations in the L1, L2, and L3 caches. These mechanisms control the caching of virtual memory pages 
and of regions of physical memory. 


° Cache control and memory ordering instructions — The Intel 64 and IA-32 architectures provide several 
instructions that control the caching of data, the ordering of memory reads and writes, and the prefetching of 
data. These instructions allow software to control the caching of specific data structures, to control memory 
coherency for specific locations in memory, and to force strong memory ordering at specific locations in a 
program. 


The following sections describe these two groups of cache control mechanisms. 


11.5.1. Cache Control Registers and Bits 


Figure 11-3 depicts cache-control mechanisms in IA-32 processors. Other than for the matter of memory address 
space, these work the same in Intel 64 processors. 


The Intel 64 and IA-32 architectures provide the following cache-control registers and bits for use in enabling or 
restricting caching to various pages or regions in memory: 


° CD flag, bit 30 of control register CRO — Controls caching of system memory locations (see Section 2.5, 
“Control Registers”). If the CD flag is clear, caching is enabled for the whole of system memory, but may be 
restricted for individual pages or regions of memory by other cache-control mechanisms. When the CD flag is 
set, caching is restricted in the processor’s caches (cache hierarchy) for the P6 and more recent processor 
families and prevented for the Pentium processor (see note below). With the CD flag set, however, the caches 
will still respond to snoop traffic. Caches should be explicitly flushed to ensure memory coherency. For highest 
processor performance, both the CD and the NW flags in control register CRO should be cleared. Table 11-5 
shows the interaction of the CD and NW flags. 


The effect of setting the CD flag is somewhat different for processor families starting with P6 family than the 
Pentium processor (see Table 11-5). To ensure memory coherency after the CD flag is set, the caches should 
be explicitly flushed (see Section 11.5.3, “Preventing Caching”). Setting the CD flag for the P6 and more 
recent processor families modifies cache line fill and update behavior. Also, setting the CD flag on these 
processors do not force strict ordering of memory accesses unless the MTRRs are disabled and/or all memory 
is referenced as uncached (see Section 8.2.5, “Strengthening or Weakening the Memory-Ordering Model”). 
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1. G flag only available in P6 and later processor families 


2. The maximum physical address size is reported by CPUID leaf 
function 80000008H. The maximum physical address size of 
FFFFFFFFFH applies only If 36-bit physical addressing is used. 

3. MTRRs available only in P6 and later processor families; 
similar control available in Pentium processor with the KEN# 

and WB/WT# pins. 

4. PAT available only in Pentium Ill and later processor families. 

5. L3 in processors based on Intel NetBurst microarchitecture can 
be disabled using IA32_MISC_ENABLE MSR. 


Figure 11-3. Cache-Control Registers and Bits Available in Intel 64 and IA-32 Processors 
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Table 11-5. Cache Operating Modes 


CD | NW Caching and Read/Write Policy 1 L2/L3! 
0 QO | Normal Cache Mode. Highest performance cache operation. 
» Read hits access the cache; read misses may cause replacement. Yes Yes 
« Write hits update the cache. Yes Yes 
* Only writes to shared lines and write misses update system memory. Yes Yes 
« Write misses cause cache line fills. Yes Yes 
« Write hits can change shared lines to modified under control of the MTRRs and with associated Yes 
read invalidation cycle. 
* (Pentium processor only.) Write misses do not cause cache line fills. Yes 
« (Pentium processor only.) Write hits can change shared lines to exclusive under control of WB/WT#. | Yes 
* Invalidation is allowed. 
» €xternal snoop traffic is supported. Yes Yes 
Yes Yes 
0 1 | Invalid setting. 
Generates a general-protection exception (HGP) with an error code of 0. NA NA 
1 QO | No-fill Cache Mode. Memory coherency is maintained.? 
« (Pentium 4 and later processor families.) State of processor after a power up or reset. Yes Yes 
* Read hits access the cache; read misses do not cause replacement (see Pentium 4 and Intel Xeon 
processors reference below). Yes Yes 
« Write hits update the cache. 
* Only writes to shared lines and write misses update system memory. 
Yes Yes 
Yes Yes 
« Write misses access memory. Yes Yes 
« Write hits can change shared lines to exclusive under control of the MTRRs and with associated Yes Yes 
read invalidation cycle. 
« (Pentium processor only.) Write hits can change shared lines to exclusive under control of the Yes 
WB/WTH#. 
« (P6 and later processor families only.) Strict memory ordering is not enforced unless the MTRRs are Yes Yes 
disabled and/or all memory is referenced as uncached (see Section 7.2.4., “Strengthening or 
Weakening the Memory Ordering Model"). 
* Invalidation is allowed. 
* €xternal snoop traffic is supported. Ves Yes 
Yes Yes 
1 1 | Memory coherency is not maintained.” 3 
« (P6 family and Pentium processors.) State of the processor after a power up or reset. Yes Yes 
« Read hits access the cache; read misses do not cause replacement. 
« Write hits update the cache and change exclusive lines to modified. Yes Yes 
Yes Yes 
« Shared lines remain shared after write hit. Yes Yes 
« Write misses access memory. Yes Yes 
*  Invalidation is inhibited when snooping; but is allowed with INVD and WBINVD instructions. Yes Yes 
* €xternal snoop traffic is supported. 
No Yes 


NOTES: 


1. The L2/L3 column in this table is definitive for the Pentium 4, Intel Xeon, and P6 family processors. It is intended to represent what 


could be implemented in a system based on a Pentium processor with an external, platform specific, write-back L2 cache. 


2. The Pentium 4 and more recent processor families do not support this mode; setting the CD and NW bits to 1 selects the no-fill 


cache mode. 


3. Not supported In Intel Atom processors. If CD = 1 in an Intel Atom processor, caching is disabled. 
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NW flag, bit 29 of control register CRO — Controls the write policy for system memory locations (see 
Section 2.5, “Control Registers”). If the NW and CD flags are clear, write-back is enabled for the whole of 
system memory, but may be restricted for individual pages or regions of memory by other cache-control 
mechanisms. Table 11-5 shows how the other combinations of CD and NW flags affects caching. 


NOTES 


For the Pentium 4 and Intel Xeon processors, the NW flag is a don’t care flag; that is, when the CD 
flag is set, the processor uses the no-fill cache mode, regardless of the setting of the NW flag. 


For Intel Atom processors, the NW flag is a don’t care flag; that is, when the CD flag is set, the 
processor disables caching, regardless of the setting of the NW flag. 


For the Pentium processor, when the L1 cache is disabled (the CD and NW flags in control register 
CRO are set), external snoops are accepted in DP (dual-processor) systems and inhibited in unipro- 
cessor systems. 


When snoops are inhibited, address parity is not checked and APCHK# is not asserted for a corrupt 
address; however, when snoops are accepted, address parity is checked and APCHK# is asserted 
for corrupt addresses. 


PCD and PWT flags in paging-structure entries — Control the memory type used to access paging 
structures and pages (see Section 4.9, “Paging and Memory Typing”). 


PCD and PWT flags in control register CR3 — Control the memory type used to access the first paging 
structure of the current paging-structure hierarchy (see Section 4.9, “Paging and Memory Typing”). 


G (global) flag in the page-directory and page-table entries (introduced to the IA-32 architecture in 
the P6 family processors) — Controls the flushing of TLB entries for individual pages. See Section 4.10, 
“Caching Translation Information,” for more information about this flag. 


PGE (page global enable) flag in control register CR4 — Enables the establishment of global pages with 
the G flag. See Section 4.10, “Caching Translation Information,” for more information about this flag. 


Memory type range registers (MTRRs) (introduced in P6 family processors) — Control the type of 
caching used in specific regions of physical memory. Any of the caching types described in Section 11.3, 
“Methods of Caching Available,” can be selected. See Section 11.11, “Memory Type Range Registers (MTRRs),” 
for a detailed description of the MTRRs. 


Page Attribute Table (PAT) MSR (introduced in the Pentium Ill processor) — Extends the memory 
typing capabilities of the processor to permit memory types to be assigned on a page-by-page basis (see 
Section 11.12, “Page Attribute Table (PAT)”). 


Third-Level Cache Disable flag, bit 6 of the IA32_MISC_ENABLE MSR (Available only in processors 
based on Intel NetBurst microarchitecture) — Allows the L3 cache to be disabled and enabled, indepen- 
dently of the L1 and L2 caches. 


KEN# and WB/WT# pins (Pentium processor) — Allow external hardware to control the caching method 
used for specific areas of memory. They perform similar (but not identical) functions to the MTRRs in the P6 
family processors. 


PCD and PWT pins (Pentium processor) — These pins (which are associated with the PCD and PWT flags in 
control register CR3 and in the page-directory and page-table entries) permit caching in an external L2 cache 
to be controlled on a page-by-page basis, consistent with the control exercised on the L1 cache of these 
processors. The P6 and more recent processor families do not provide these pins because the L2 cache in 
internal to the chip package. 


11.5.2 Precedence of Cache Controls 


The cache control flags and MTRRs operate hierarchically for restricting caching. That is, if the CD flag is set, 

caching is prevented globally (see Table 11-5). If the CD flag is clear, the page-level cache control flags and/or the 
MTRRs can be used to restrict caching. If there is an overlap of page-level and MTRR caching controls, the mecha- 
nism that prevents caching has precedence. For example, if an MTRR makes a region of system memory uncache- 
able, a page-level caching control cannot be used to enable caching for a page in that region. The converse is also 
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true; that is, if a page-level caching control designates a page as uncacheable, an MTRR cannot be used to make 
the page cacheable. 


In cases where there is a overlap in the assignment of the write-back and write-through caching policies to a page 
and a region of memory, the write-through policy takes precedence. The write-combining policy (which can only be 
assigned through an MTRR or the PAT) takes precedence over either write-through or write-back. 


The selection of memory types at the page level varies depending on whether PAT is being used to select memory 
types for pages, as described in the following sections. 


On processors based on Intel NetBurst microarchitecture, the third-level cache can be disabled by bit 6 of the 
IA32_MISC_ENABLE MSR. Using IA32_MISC_ENABLE[bit 6] takes precedence over the CD flag, MTRRs, and PAT 
for the L3 cache in those processors. That is, when the third-level cache disable flag is set (cache disabled), the 
other cache controls have no affect on the L3 cache; when the flag is clear (enabled), the cache controls have the 
same affect on the L3 cache as they have on the L1 and L2 caches. 


IA32_MISC_ENABLE[bit 6] is not supported in Intel Core i7 processors, nor processors based on Intel Core, and 
Intel Atom microarchitectures. 


11.5.2.1 Selecting Memory Types for Pentium Pro and Pentium II Processors 


The Pentium Pro and Pentium II processors do not support the PAT. Here, the effective memory type for a page is 
selected with the MTRRs and the PCD and PWT bits in the page-table or page-directory entry for the page. Table 
11-6 describes the mapping of MTRR memory types and page-level caching attributes to effective memory types, 
when normal caching is in effect (the CD and NW flags in control register CRO are clear). Combinations that appear 
in gray are implementation-defined for the Pentium Pro and Pentium II processors. System designers are encour- 
aged to avoid these implementation-defined combinations. 


Table 11-6. Effective Page-Level Memory Type for Pentium Pro and Pentium II Processors 


MTRR Memory Type! PCD Value PWT Value Effective Memory Type 
UC X X UC 
WC 0 0 WC 
0 1 WC 
1 0 WC 
1 1 UC 
WT 0 xX WT 
1 X UC 
WP 0 0 WP 
0 1 WP 
1 0 WC 
1 1 UC 
WB 0 0 WB 
0 1 WT 
1 X UC 
NOTE: 


1. These effective memory types also apply to the Pentium 4, Intel Xeon, and Pentium Ill processors when the PAT bit is not used 
(set to 0) in page-table and page-directory entries. 


When normal caching is in effect, the effective memory type shown in Table 11-6 is determined using the following 
rules: 


1. If the PCD and PWT attributes for the page are both O, then the effective memory type is identical to the 
MTRR-defined memory type. 
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If the PCD flag is set, then the effective memory type is UC. 


If the PCD flag is clear and the PWT flag is set, the effective memory type is WT for the WB memory type and 
the MTRR-defined memory type for all other memory types. 


4. Setting the PCD and PWT flags to opposite values is considered model-specific for the WP and WC memory 
types and architecturally-defined for the WB, WT, and UC memory types. 


11.5.2.2 Selecting Memory Types for Pentium III and More Recent Processor Families 


The Intel Core 2 Duo, Intel Atom, Intel Core Duo, Intel Core Solo, Pentium M, Pentium 4, Intel Xeon, and Pentium 
lll processors use the PAT to select effective page-level memory types. Here, a memory type for a page is selected 
by the MTRRs and the value in a PAT entry that is selected with the PAT, PCD and PWT bits in a page-table or page- 
directory entry (see Section 11.12.3, “Selecting a Memory Type from the PAT”). Table 11-7 describes the mapping 
of MTRR memory types and PAT entry types to effective memory types, when normal caching is in effect (the CD 
and NW flags in control register CRO are clear). 


Table 11-7. Effective Page-Level Memory Types for Pentium III and More Recent Processor Families 


MTRR Memory Type PAT Entry Value Effective Memory Type 
UC UC uc! 
UC- uc! 
Wc wc 
WT UC! 
WB uc! 
WP uc! 
wc UC UCZ 
UC- wc 
Wc wc 
WT uc#3 
WB wc 
WP uc#3 
WT UC We 
UC- UC2 
wc wc 
WT WT 
WB WT 
WP wp? 
WB UC UCZ 
UC- UCZ 
wc wc 
WT WT 
WB WB 
WP WP 
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Table 11-7. Effective Page-Level Memory Types for Pentium III and More Recent Processor Families (Contd.) 


MTRR Memory Type PAT Entry Value Effective Memory Type 
WP UC UC? 
UC- we 
WC WC 
WT wT? 
WB WP 
WP WP 


NOTES: 

1. The UC attribute comes from the MTRRs and the processors are not required to snoop their caches since the data could never have 
been cached. This attribute is preferred for performance reasons. 

2. The UC attribute came from the page-table or page-directory entry and processors are required to check their caches because the 
data may be cached due to page aliasing, which is not recommended. 

3. These combinations were specified as “undefined” in previous editions of the Intel® 64 and IA-32 Architectures Software Devel- 
oper’s Manual. However, all processors that support both the PAT and the MTRRs determine the effective page-level memory 
types for these combinations as given. 


11.5.2.3. Writing Values Across Pages with Different Memory Types 


If two adjoining pages in memory have different memory types, and a word or longer operand is written to a 
memory location that crosses the page boundary between those two pages, the operand might be written to 
memory twice. This action does not present a problem for writes to actual memory; however, if a device is mapped 
the memory space assigned to the pages, the device might malfunction. 


11.5.3. Preventing Caching 


To disable the L1, L2, and L3 caches after they have been enabled and have received cache fills, perform the 
following steps: 


1. Enter the no-fill cache mode. (Set the CD flag in control register CRO to 1 and the NW flag to O. 
2. Flush all caches using the WBINVD instruction. 


3. Disable the MTRRs and set the default memory type to uncached or set all MTRRs for the uncached memory 
type (see the discussion of the discussion of the TYPE field and the E flag in Section 11.11.2.1, 
“IA32_MTRR_DEF_TYPE MSR"). 


The caches must be flushed (step 2) after the CD flag is set to ensure system memory coherency. If the caches are 
not flushed, cache hits on reads will still occur and data will be read from valid cache lines. 


The intent of the three separate steps listed above address three distinct requirements: (i) discontinue new data 
replacing existing data in the cache (ii) ensure data already in the cache are evicted to memory, (iii) ensure subse- 
quent memory references observe UC memory type semantics. Different processor implementation of caching 
control hardware may allow some variation of software implementation of these three requirements. See note 
below. 


NOTES 


Setting the CD flag in control register CRO modifies the processor’s caching behavior as indicated in 
Table 11-5, but setting the CD flag alone may not be sufficient across all processor families to force 
the effective memory type for all physical memory to be UC nor does it force strict memory 
ordering, due to hardware implementation variations across different processor families. To force 
the UC memory type and strict memory ordering on all of physical memory, it is sufficient to either 
program the MTRRs for all physical memory to be UC memory type or disable all MTRRs. 


For the Pentium 4 and Intel Xeon processors, after the sequence of steps given above has been 
executed, the cache lines containing the code between the end of the WBINVD instruction and 
before the MTRRS have actually been disabled may be retained in the cache hierarchy. Here, to 
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remove code from the cache completely, a second WBINVD instruction must be executed after the 
MTRRs have been disabled. 


For Intel Atom processors, setting the CD flag forces all physical memory to observe UC semantics 
(without requiring memory type of physical memory to be set explicitly). Consequently, software 
does not need to issue a second WBINVD as some other processor generations might require. 


11.5.4 Disabling and Enabling the L3 Cache 


On processors based on Intel NetBurst microarchitecture, the third-level cache can be disabled by bit 6 of the 
IA32_MISC_ENABLE MSR. The third-level cache disable flag (bit 6 of the IA32_MISC_ENABLE MSR) allows the L3 
cache to be disabled and enabled, independently of the L1 and L2 caches. Prior to using this control to disable or 
enable the L3 cache, software should disable and flush all the processor caches, as described earlier in Section 
11.5.3, “Preventing Caching,” to prevent of loss of information stored in the L3 cache. After the L3 cache has been 
disabled or enabled, caching for the whole processor can be restored. 


Newer Intel 64 processor with L3 do not support IA32_MISC_ENABLE[bit 6], the procedure described in Section 
11.5.3, “Preventing Caching,” apply to the entire cache hierarchy. 


11.5.5 Cache Management Instructions 


The Intel 64 and IA-32 architectures provide several instructions for managing the Li, L2, and L3 caches. The INVD 
and WBINVD instructions are privileged instructions and operate on the Li, L2 and L3 caches as a whole. The 
PREFETCHh, CLFLUSH and CLFLUSHOPT instructions and the non-temporal move instructions (MOVNTI, MOVNTQ, 
MOVNTDQ, MOVNTPS, and MOVNTPD) offer more granular control over caching, and are available to all privileged 
levels. 


The INVD and WBINVD instructions are used to invalidate the contents of the L1, L2, and L3 caches. The INVD 
instruction invalidates all internal cache entries, then generates a special-function bus cycle that indicates that 
external caches also should be invalidated. The INVD instruction should be used with care. It does not force a 
write-back of modified cache lines; therefore, data stored in the caches and not written back to system memory 
will be lost. Unless there is a specific requirement or benefit to invalidating the caches without writing back the 
modified lines (such as, during testing or fault recovery where cache coherency with main memory is not a 
concern), software should use the WBINVD instruction. 


The WBINVD instruction first writes back any modified lines in all the internal caches, then invalidates the contents 
of both the L1, L2, and L3 caches. It ensures that cache coherency with main memory is maintained regardless of 
the write policy in effect (that is, write-through or write-back). Following this operation, the WBINVD instruction 
generates one (P6 family processors) or two (Pentium and Intel486 processors) special-function bus cycles to indi- 
cate to external cache controllers that write-back of modified data followed by invalidation of external caches 
should occur. The amount of time or cycles for WBINVD to complete will vary due to the size of different cache hier- 
archies and other factors. As a consequence, the use of the WBINVD instruction can have an impact on inter- 
rupt/event response time. 


The PREFETCHA instructions allow a program to suggest to the processor that a cache line from a specified location 
in system memory be prefetched into the cache hierarchy (see Section 11.8, “Explicit Caching”). 


The CLFLUSH and CLFLUSHOPT instructions allow selected cache lines to be flushed from memory. These instruc- 
tions give a program the ability to explicitly free up cache space, when it is known that cached section of system 
memory will not be accessed in the near future. 


The non-temporal move instructions (MOVNTI, MOVNTQ, MOVNTDQ, MOVNTPS, and MOVNTPD) allow data to be 
moved from the processor's registers directly into system memory without being also written into the L1, L2, 
and/or L3 caches. These instructions can be used to prevent cache pollution when operating on data that is going 
to be modified only once before being stored back into system memory. These instructions operate on data in the 
general-purpose, MMX, and XMM registers. 
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11.5.6 1 Data Cache Context Mode 


Li data cache context mode is a feature of processors based on the Intel NetBurst microarchitecture that support 
Intel Hyper-Threading Technology. When CPUID.1:ECX[bit 10] = 1, the processor supports setting L1 data cache 
context mode using the L1 data cache context mode flag ( IA32_MISC_ENABLE[bit 24] ). Selectable modes are 
adaptive mode (default) and shared mode. 


The BIOS is responsible for configuring the L1 data cache context mode. 


11.5.6.1 Adaptive Mode 


Adaptive mode facilitates L1 data cache sharing between logical processors. When running in adaptive mode, the 
L1 data cache is shared across logical processors in the same core if: 


® CR3 control registers for logical processors sharing the cache are identical. 
¢ The same paging mode is used by logical processors sharing the cache. 


In this situation, the entire L1 data cache is available to each logical processor (instead of being competitively 
shared). 


If CR3 values are different for the logical processors sharing an Li data cache or the logical processors use different 
paging modes, processors compete for cache resources. This reduces the effective size of the cache for each logical 
processor. Aliasing of the cache is not allowed (which prevents data thrashing). 


11.5.6.2 Shared Mode 


In shared mode, the L1 data cache is competitively shared between logical processors. This is true even if the 
logical processors use identical CR3 registers and paging modes. 


In shared mode, linear addresses in the L1 data cache can be aliased, meaning that one linear address in the cache 
can point to different physical locations. The mechanism for resolving aliasing can lead to thrashing. For this 
reason, IA32_MISC_ENABLE[bit 24] = 0 is the preferred configuration for processors based on the Intel NetBurst 
microarchitecture that support Intel Hyper-Threading Technology. 


11.6 SELF-MODIFYING CODE 


A write to a memory location in a code segment that is currently cached in the processor causes the associated 
cache line (or lines) to be invalidated. This check is based on the physical address of the instruction. In addition, 
the P6 family and Pentium processors check whether a write to a code segment may modify an instruction that has 
been prefetched for execution. If the write affects a prefetched instruction, the prefetch queue is invalidated. This 
latter check is based on the linear address of the instruction. For the Pentium 4 and Intel Xeon processors, a write 
or a snoop of an instruction in a code segment, where the target instruction is already decoded and resident in the 
trace cache, invalidates the entire trace cache. The latter behavior means that programs that self-modify code can 
cause severe degradation of performance when run on the Pentium 4 and Intel Xeon processors. 


In practice, the check on linear addresses should not create compatibility problems among IA-32 processors. Appli- 
cations that include self-modifying code use the same linear address for modifying and fetching the instruction. 
Systems software, such as a debugger, that might possibly modify an instruction using a different linear address 
than that used to fetch the instruction, will execute a serializing operation, such as a CPUID instruction, before the 
modified instruction is executed, which will automatically resynchronize the instruction cache and prefetch queue. 
(See Section 8.1.3, “Handling Self- and Cross-Modifying Code,” for more information about the use of self-modi- 
fying code.) 


For Intel486 processors, a write to an instruction in the cache will modify it in both the cache and memory, but if 
the instruction was prefetched before the write, the old version of the instruction could be the one executed. To 
prevent the old instruction from being executed, flush the instruction prefetch unit by coding a jump instruction 
immediately after any write that modifies an instruction. 
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11.7. IMPLICIT CACHING (PENTIUM 4, INTEL XEON, 
AND P6 FAMILY PROCESSORS) 


Implicit caching occurs when a memory element is made potentially cacheable, although the element may never 
have been accessed in the normal von Neumann sequence. Implicit caching occurs on the P6 and more recent 
processor families due to aggressive prefetching, branch prediction, and TLB miss handling. Implicit caching is an 
extension of the behavior of existing Intel386, Intel486, and Pentium processor systems, since software running 
on these processor families also has not been able to deterministically predict the behavior of instruction prefetch. 


To avoid problems related to implicit caching, the operating system must explicitly invalidate the cache when 
changes are made to cacheable data that the cache coherency mechanism does not automatically handle. This 
includes writes to dual-ported or physically aliased memory boards that are not detected by the snooping mecha- 
nisms of the processor, and changes to page- table entries in memory. 


The code in Example 11-1 shows the effect of implicit caching on page-table entries. The linear address FOOOH 
points to physical location BOOOH (the page-table entry for FOOOH contains the value BOOOH), and the page-table 
entry for linear address FOOO is PTE_FOOO. 


Example 11-1. Effect of Implicit Caching on Page-Table Entries 


mov EAX, CR3; Invalidate the TLB 

mov CR3, EAX; by copying CR3 to itself 

mov PTE_FOOO, AOOOH; Change FOOOH to point to AOOOH 
mov EBX, [FOOOH]; 


Because of speculative execution in the P6 and more recent processor families, the last MOV instruction performed 
would place the value at physical location BOOOH into EBX, rather than the value at the new physical address 
AOOOH. This situation is remedied by placing a TLB invalidation between the load and the store. 


11.8 EXPLICIT CACHING 


The Pentium III processor introduced four new instructions, the PREFETCHA instructions, that provide software with 
explicit control over the caching of data. These instructions provide “hints” to the processor that the data requested 
by a PREFETCHA instruction should be read into cache hierarchy now or as soon as possible, in anticipation of its 
use. The instructions provide different variations of the hint that allow selection of the cache level into which data 
will be read. 


The PREFETCHA instructions can help reduce the long latency typically associated with reading data from memory 
and thus help prevent processor “stalls.” However, these instructions should be used judiciously. Overuse can lead 
to resource conflicts and hence reduce the performance of an application. Also, these instructions should only be 
used to prefetch data from memory; they should not be used to prefetch instructions. For more detailed informa- 
tion on the proper use of the prefetch instruction, refer to Chapter 7, “Optimizing Cache Usage,” in the Inte/® 64 
and IA-32 Architectures Optimization Reference Manual. 


11.9. INVALIDATING THE TRANSLATION LOOKASIDE BUFFERS (TLBS) 


The processor updates its address translation caches (TLBs) transparently to software. Several mechanisms are 
available, however, that allow software and hardware to invalidate the TLBs either explicitly or as a side effect of 
another operation. Most details are given in Section 4.10.4, “Invalidation of TLBs and Paging-Structure Caches.” In 
addition, the following operations invalidate all TLB entries, irrespective of the setting of the G flag: 


¢ Asserting or de-asserting the FLUSH# pin. 
¢ (Pentium 4, Intel Xeon, and later processors only.) Writing to an MTRR (with a WRMSR instruction). 
¢ Writing to control register CRO to modify the PG or PE flag. 
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e (Pentium 4, Intel Xeon, and later processors only.) Writing to control register CR4 to modify the PSE, PGE, or 
PAE flag. 


* Writing to control register CR4 to change the PCIDE flag from 1 to 0. 
See Section 4.10, “Caching Translation Information,” for additional information about the TLBs. 


11.10 STORE BUFFER 


Intel 64 and IA-32 processors temporarily store each write (store) to memory in a store buffer. The store buffer 
improves processor performance by allowing the processor to continue executing instructions without having to 
wait until a write to memory and/or to a cache is complete. It also allows writes to be delayed for more efficient use 
of memory-access bus cycles. 


In general, the existence of the store buffer is transparent to software, even in systems that use multiple proces- 
sors. The processor ensures that write operations are always carried out in program order. It also ensures that the 
contents of the store buffer are always drained to memory in the following situations: 


¢ When an exception or interrupt is generated. 

¢ (P6 and more recent processor families only) When a serializing instruction is executed. 

¢ When an J/O instruction is executed. 

¢ When a LOCK operation is performed. 

¢ (P6 and more recent processor families only) When a BINIT operation is performed. 

¢ (Pentium Ill, and more recent processor families only) When using an SFENCE instruction to order stores. 
¢ (Pentium 4 and more recent processor families only) When using an MFENCE instruction to order stores. 


The discussion of write ordering in Section 8.2, “Memory Ordering,” gives a detailed description of the operation of 
the store buffer. 


11.11 MEMORY TYPE RANGE REGISTERS (MTRRS) 


The following section pertains only to the P6 and more recent processor families. 


The memory type range registers (MTRRs) provide a mechanism for associating the memory types (see Section 
11.3, “Methods of Caching Available”) with physical-address ranges in system memory. They allow the processor to 
optimize operations for different types of memory such as RAM, ROM, frame-buffer memory, and memory-mapped 
I/O devices. They also simplify system hardware design by eliminating the memory control pins used for this func- 
tion on earlier IA-32 processors and the external logic needed to drive them. 


The MTRR mechanism allows multiple ranges to be defined in physical memory, and it defines a set of model- 
specific registers (MSRs) for specifying the type of memory that is contained in each range. Table 11-8 shows the 
memory types that can be specified and their properties; Figure 11-4 shows the mapping of physical memory with 
MTRRs. See Section 11.3, “Methods of Caching Available,” for a more detailed description of each memory type. 


Following a hardware reset, the P6 and more recent processor families disable all the fixed and variable MTRRs, 
which in effect makes all of physical memory uncacheable. Initialization software should then set the MTRRs to a 
specific, system-defined memory map. Typically, the BIOS (basic input/output system) software configures the 
MTRRs. The operating system or executive is then free to modify the memory map using the normal page-level 
cacheability attributes. 


In a multiprocessor system using a processor in the P6 family or a more recent family, each processor MUST use 
the identical MTRR memory map so that software will have a consistent view of memory. 


NOTE 


In multiple processor systems, the operating system must maintain MTRR consistency between all 
the processors in the system (that is, all processors must use the same MTRR values). The P6 and 
more recent processor families provide no hardware support for maintaining this consistency. 
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Table 11-8. Memory Types That Can Be Encoded in MTRRs 


Memory Type and Mnemonic Encoding in MTRR 
Uncacheable (UC) OOH 
Write Combining (WC) 01H 
Reserved* 02H 
Reserved* 03H 
Write-through (WT) 04H 
Write-protected (WP) O5H 
Writeback (WB) O6H 
Reserved* 7H through FFH 
NOTE: 
* Use of these encodings results in a general-protection exception (#GP). 
Physical Memory 
FFFFFFFFH 
Address ranges not 
mapped by an MTRR —_»S Ss 
are set to a default type 
Variable ranges 
(from 4KBytesto “~ {[________] 
maximum size of 
physicalmemory) \\ |__~_~_~____4J] 
100000H 
64 fixed ranges FFFFFH 
(4 KBytes each) 256 KBytes C0000H 
16 fixed ranges BFFFFH 
(16 KBytes each) | 26 KBytes sang 
7FFFFH 
8 fixed ranges 
(64-KBytes each) |_~—«O12 KBytes 
0 


Figure 11-4. Mapping Physical Memory With MTRRs 


11.11.1 MTRR Feature Identification 


The availability of the MTRR feature is model-specific. Software can determine if MTRRs are supported ona 
processor by executing the CPUID instruction and reading the state of the MTRR flag (bit 12) in the feature infor- 


mation register (EDX). 


If the MTRR flag is set (indicating that the processor implements MTRRs), additional information about MTRRs can 
be obtained from the 64-bit IA32_MTRRCAP MSR (named MTRRcap MSR for the P6 family processors). The 
IA32_MTRRCAP MSR is a read-only MSR that can be read with the RDMSR instruction. Figure 11-5 shows the 
contents of the IA32_MTRRCAP MSR. The functions of the flags and field in this register are as follows: 


Vol. 3A 11-21 


MEMORY CACHE CONTROL 


¢ VCNT (variable range registers count) field, bits O through 7 — Indicates the number of variable ranges 
implemented on the processor. 

° FIX (fixed range registers supported) flag, bit 8 — Fixed range MTRRs (IA32_MTRR_FIX64K_00000 
through IA32_MTRR_FIX4K_OF8000) are supported when set; no fixed range registers are supported when 
clear. 


¢ WC (write combining) flag, bit 10 — The write-combining (WC) memory type is supported when set; the 
WC type is not supported when clear. 


° SMRR (System-Management Range Register) flag, bit 11 — The system-management range register 
(SMRR) interface is supported when bit 11 is set; the SMRR interface is not supported when clear. 


Bit 9 and bits 12 through 63 in the IA32_MTRRCAP MSR are reserved. If software attempts to write to the 
IA32_MTRRCAP MSR, a general-protection exception (#GP) is generated. 


Software must read IA32_MTRRCAP VCNT field to determine the number of variable MTRRs and query other 
feature bits in IA32_MTRRCAP to determine additional capabilities that are supported in a processor. For example, 
some processors may report a value of ‘8’ in the VCNT field, other processors may report VCNT with different 
values. 


63 11109 87 0 


Ww 
Cc 


F 
Reserved | : VCNT 


SMRR — SMRR interface supported | 


WC — Write-combining memory type supported — 
FIX — Fixed range registers supported 
VCNT — Number of variable range registers 


Reserved 


Figure 11-5. IA32_MTRRCAP Register 


11.11.2 Setting Memory Ranges with MTRRs 


The memory ranges and the types of memory specified in each range are set by three groups of registers: the 
IA32_MTRR_DEF_TYPE MSR, the fixed-range MTRRs, and the variable range MTRRs. These registers can be read 
and written to using the RDMSR and WRMSR instructions, respectively. The IA32_MTRRCAP MSR indicates the 
availability of these registers on the processor (see Section 11.11.1, “MTRR Feature Identification”). 


11.11.2.1 IA32_MTRR_DEF_TYPE MSR 


The IA32_MTRR_DEF_TYPE MSR (named MTRRdefType MSR for the P6 family processors) sets the default proper- 
ties of the regions of physical memory that are not encompassed by MTRRs. The functions of the flags and field in 
this register are as follows: 


° Type field, bits O through 7 — Indicates the default memory type used for those physical memory address 
ranges that do not have a memory type specified for them by an MTRR (see Table 11-8 for the encoding of this 
field). The legal values for this field are 0, 1, 4, 5, and 6. All other values result in a general-protection 
exception (#GP) being generated. 


Intel recommends the use of the UC (uncached) memory type for all physical memory addresses where 
memory does not exist. To assign the UC type to nonexistent memory locations, it can either be specified as the 
default type in the Type field or be explicitly assigned with the fixed and variable MTRRs. 
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63 1211109 87 0 


Reserved E Type 


E — MTRR enable/disable 
FE — Fixed-range MTRRs enable/disable —— 
Type — Default memory type 


Reserved 


Figure 11-6. [A32_MTRR_DEF_TYPE MSR 


¢ FE (fixed MTRRs enabled) flag, bit 10 — Fixed-range MTRRs are enabled when set; fixed-range MTRRs are 
disabled when clear. When the fixed-range MTRRs are enabled, they take priority over the variable-range 
MTRRs when overlaps in ranges occur. If the fixed-range MTRRs are disabled, the variable-range MTRRs can 
still be used and can map the range ordinarily covered by the fixed-range MTRRs. 

¢ E(MTRRs enabled) flag, bit 11 — MTRRs are enabled when set; all MTRRs are disabled when clear, and the 
UC memory type is applied to all of physical memory. When this flag is set, the FE flag can disable the fixed- 
range MTRRs; when the flag is clear, the FE flag has no affect. When the E flag is set, the type specified in the 
default memory type field is used for areas of memory not already mapped by either a fixed or variable MTRR. 


Bits 8 and 9, and bits 12 through 63, in the IA32_MTRR_DEF_TYPE MSR are reserved; the processor generates a 
general-protection exception (#GP) if software attempts to write nonzero values to them. 


11.11.2.2 Fixed Range MTRRs 

The fixed memory ranges are mapped with 11 fixed-range registers of 64 bits each. Each of these registers is 

divided into 8-bit fields that are used to specify the memory type for each of the sub-ranges the register controls: 

° Register IA32_MTRR_FIX64K_00000 — Maps the 512-KByte address range from OH to 7FFFFH. This range 
is divided into eight 64-KByte sub-ranges. 


° Registers IA32_MTRR_FIX16K_80000 and IA32_MTRR_FIX16K_AO0000 — Maps the two 128-KByte 
address ranges from 80000H to BFFFFH. This range is divided into sixteen 16-KByte sub-ranges, 8 ranges per 
register. 


° Registers IA32_MTRR_FIX4K_C0000 through IA32_MTRR_FIX4K_F8000 — Maps eight 32-KByte 
address ranges from COOOOH to FFFFFH. This range is divided into sixty-four 4-KByte sub-ranges, 8 ranges per 
register. 


Table 11-9 shows the relationship between the fixed physical-address ranges and the corresponding fields of the 
fixed-range MTRRs; Table 11-8 shows memory type encoding for MTRRs. 


For the P6 family processors, the prefix for the fixed range MTRRs is MTRRfix. 


11.11.2.3 Variable Range MTRRs 


The Pentium 4, Intel Xeon, and P6 family processors permit software to specify the memory type for m variable- 
size address ranges, using a pair of MTRRs for each range. The number m of ranges supported is given in bits 7:0 
of the IA32_MTRRCAP MSR (see Figure 11-5 in Section 11.11.1). 


The first entry in each pair (IA32_MTRR_PHYSBASEn) defines the base address and memory type for the range; 
the second entry (IA32_MTRR_PHYSMASKn) contains a mask used to determine the address range. The “n” suffix 
is in the range 0 through m-1 and identifies a specific register pair. 


For P6 family processors, the prefixes for these variable range MTRRs are MTRRphysBase and MTRRphysMask. 
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Table 11-9. Address Mapping for Fixed-Range MTRRs 


Address Range (hexadecimal) MTRR 

63 56 55 48 47 40 39 32 31 24 23 «16 15 8 7 0 

70000- 60000- 50000- 40000- 30000- 20000- 10000- 00000- IA32_MTRR_ 
7FFFF 6FFFF 5SFFFF 4FFFF 3FFFF 2FFFF 1FFFF OFFFF FIX64K_00000 
9CO00 98000- 94000- 90000- 8C000- 88000- 84000- 80000- IA32_MTRR_ 
OFFFF OBFFF 97FFF 93FFF 8FFFF 8BFFF 87FFF 83FFF FIX16K_80000 
BCOOO B8000- B4000- BOOOO- ACOO0- A8000- A4000- AOQO00- IA32_MTRR_ 
BFFFF BBFFF B7FFF B3FFF AFFFF ABFFF A7FFF A3FFF FIX16K_A0000 
C7000 C6000- C5000- C4000- C3000- C2000- C1000- Co000- IA32_MTRR_ 
C7FFF C6FFF CSFFF C4FFF C3FFF C2FFF C1FFF COFFF FIX4K_COO00 
CFOOO CEOO0- CDO00- CCOO0- CBOO0- CAO00- C9000- C8000- IA32_MTRR_ 
CFFFF CEFFF CDFFF CCFFF CBFFF CAFFF COFFF C8FFF FIX4K_C8000 
D7000 D6000- D5000- D4000- D3000- D2000- D1000- DOOOD- IA32_MTRR_ 
D7FFF D6FFF D5FFF D4FFF D3FFF D2FFF D1FFF DOFFF FIX4K_DO000 
DFOOO DEOOO- DDOOO- DCOOO- DBOOO- DAOOO- D9000- D8000- IA32_MTRR_ 
DFFFF DEFFF DDFFF DCFFF DBFFF DAFFF DOFFF D8FFF FIX4K_D8000 
€7000 €6000- €5000- €4000- €3000- €2000- €1000- €0000- IA32_MTRR_ 
E7FFF E6FFF ESFFF E4FFF E3FFF E2FFF E1FFF EOFFF FIX4K_E0000 
EFOOO EEOOO- EDOOO- ECOOO- EBOOO- EAOOO- €9000- €8000- IA32_MTRR_ 
EFFFF EEFFF EDFFF ECFFF EBFFF EAFFF ESFFF E8FFF FIX4K_E8000 
F7000 F6000- F5000- F4000- F3000- F2000- F1000- FOOOO- IA32_MTRR_ 
F7FFF F6FFF FSFFF F4FFF F3FFF F2FFF FIFFF FOFFF FIX4K_FOO00 
FFOOO FEOOO- FDOOO- FCOOO- FBOOO- FAQOO- F9000- F8000- IA32_MTRR_ 
FFFFF FEFFF FDFFF FCFFF FBFFF FAFFF FOFFF F8FFF FIX4K_F8000 


Figure 11-7 shows flags and fields in these registers. The functions of these flags and fields are: 


° Type field, bits 0 through 7 — Specifies the memory type for the range (see Table 11-8 for the encoding of 


this field). 


° PhysBase field, bits 12 through (MAXPHYADDR-1) — Specifies the base address of the address range. 
This 24-bit value, in the case where MAXPHYADDR is 36 bits, is extended by 12 bits at the low end to form the 
base address (this automatically aligns the address on a 4-KByte boundary). 

¢ PhysMask field, bits 12 through (MAXPHYADDR-1) — Specifies a mask (24 bits if the maximum physical 
address size is 36 bits, 28 bits if the maximum physical address size is 40 bits). The mask determines the range 


of the region being mapped, according to the following relationships: 
— Address_Within_Range AND PhysMask = PhysBase AND PhysMask 


— This value is extended by 12 bits at the low end to form the mask value. For more information: see Section 
11.11.3, “Example Base and Mask Calculations.” 


— The width of the PhysMask field depends on the maximum physical address size supported by the 
processor. 


CPUID.80000008H reports the maximum physical address size supported by the processor. If 


CPUID.80000008H is not available, software may assume that the processor supports a 36-bit physical 


address size (then PhysMask is 24 bits wide and the upper 28 bits of IA832_MTRR_PHYSMASkKn are 
reserved). See the Note below. 


* V (valid) flag, bit 11 — Enables the register pair when set; disables register pair when clear. 
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IA32_MTRR_PHYSBASEn Register 
63 MAXPHYADDR_ 1211 87 0 


Reserved PhysBase Type 


PhysBase — Base address of range —! 
Type — Memory type for range 


IA32_MTRR_PHYSMASKn Register 
63 MAXPHYADDR 121110 0 


Reserved PhysMask Vv Reserved 


PhysMask — Sets range mask — 
V — Valid 


Reserved 


MAXPHYADDR: The bit position indicated by MAXPHYADDR depends on the maximum 
physical address range supported by the processor. It is reported by CPUID leaf 
function 80000008H. If CPUID does not support leaf 80000008H, the processor 
supports 36-bit physical address size, then bit PhysMask consists of bits 35:12, and 

bits 63:36 are reserved. 


Figure 11-7. IA32_MTRR_PHYSBASEn and IA32_MTRR_PHYSMASKn Variable-Range Register Pair 


All other bits in the IA32_MTRR_PHYSBASEn and IA32_MTRR_PHYSMASKnr registers are reserved; the processor 
generates a general-protection exception (#GP) if software attempts to write to them. 


Some mask values can result in ranges that are not continuous. In such ranges, the area not mapped by the mask 
value is set to the default memory type, unless some other MTRR specifies a type for that range. Intel does not 
encourage the use of “discontinuous” ranges. 


NOTE 


It is possible for software to parse the memory descriptions that BIOS provides by using the 
ACPI/INT15 e820 interface mechanism. This information then can be used to determine how 
MTRRs are initialized (for example: allowing the BIOS to define valid memory ranges and the 
maximum memory range supported by the platform, including the processor). 


See Section 11.11.4.1, “MTRR Precedences,” for information on overlapping variable MTRR ranges. 


11.11.2.4 System-Management Range Register Interface 


If IA32_MTRRCAP[bit 11] is set, the processor supports the SMRR interface to restrict access to a specified 
memory address range used by system-management mode (SMM) software (see Section 30.4.2.1). If the SMRR 


interface is supported, SMM software is strongly encouraged to use it to protect the SMI code and data stored by 
SMI handler in the SMRAM region. 


The system-management range registers consist of a pair of MSRs (see Figure 11-8). The IA32_SMRR_PHYSBASE 
MSR defines the base address for the SMRAM memory range and the memory type used to access it in SMM. The 
IA32_SMRR_PHYSMASK MSR contains a valid bit and a mask that determines the SMRAM address range protected 
by the SMRR interface. These MSRs may be written only in SMM; an attempt to write them outside of SMM causes 
a general-protection exception. 


Figure 11-8 shows flags and fields in these registers. The functions of these flags and fields are the following: 


1. For some processor models, these MSRs can be accessed by RDMSR and WRMSR only if the SMRR interface has been enabled using 
a model-specific bit in the [A32_FEATURE_CONTROL MSR. 
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° Type field, bits O through 7 — Specifies the memory type for the range (see Table 11-8 for the encoding of 
this field). 


¢ PhysBase field, bits 12 through 31 — Specifies the base address of the address range. The address must be 
less than 4 GBytes and is automatically aligned on a 4-KByte boundary. 


¢ PhysMask field, bits 12 through 31 — Specifies a mask that determines the range of the region being 
mapped, according to the following relationships: 


— Address_Within_Range AND PhysMask = PhysBase AND PhysMask 


— This value is extended by 12 bits at the low end to form the mask value. For more information: see Section 
11.11.3, “Example Base and Mask Calculations.” 


° V (valid) flag, bit 11 — Enables the register pair when set; disables register pair when clear. 


Before attempting to access these SMRR registers, software must test bit 11 in the IA32_MTRRCAP register. If 
SMRR is not supported, reads from or writes to registers cause general-protection exceptions. 


When the valid flag in the IA32_SMRR_PHYSMASK MSR is 1, accesses to the specified address range are treated as 
follows: 


e If the logical processor is in SMM, accesses uses the memory type in the IA32_SMRR_PHYSBASE MSR. 


° If the logical processor is not in SMM, write accesses are ignored and read accesses return a fixed value for each 
byte. The uncacheable memory type (UC) is used in this case. 


The above items apply even if the address range specified overlaps with a range specified by the MTRRs. 


IA32_SMRR_PHYSBASE Register 
63 31 1211 87 0 


Reserved PhysBase Type 


PhysBase — Base address of range — 
Type — Memory type for range 


IA32_SMRR_PHYSMASK Register 
63 31 121110 0 


Reserved PhysMask v Reserved 


PhysMask — Sets range mask — 
V — Valid 


Reserved 


Figure 11-8. [A32_SMRR_PHYSBASE and IA32_SMRR_PHYSMASK SMRR Pair 


11.11.3 Example Base and Mask Calculations 


The examples in this section apply to processors that support a maximum physical address size of 36 bits. The base 
and mask values entered in variable-range MTRR pairs are 24-bit values that the processor extends to 36-bits. 


For example, to enter a base address of 2 MBytes (200000H) in the IA32_MTRR_PHYSBASES3 register, the 12 least- 
significant bits are truncated and the value 000200H is entered in the PhysBase field. The same operation must be 
performed on mask values. For example, to map the address range from 200000H to 3FFFFFH (2 MBytes to 4 
MBytes), a mask value of FFFEQOOOOH is required. Again, the 12 least-significant bits of this mask value are trun- 
cated, so that the value entered in the PhysMask field of IA32_MTRR_PHYSMASK3 is FFFEQOH. This mask is chosen 
so that when any address in the 200000H to 3FFFFFH range is AND’d with the mask value, it will return the same 
value as when the base address is AND’d with the mask value (which is 200000H). 
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To map the address range from 400000H to 7FFFFFH (4 MBytes to 8 MBytes), a base value of 000400H is entered 
in the PhysBase field and a mask value of FFFCOOH is entered in the PhysMask field. 


Example 11-2. Setting-Up Memory for a System 

Here is an example of setting up the MTRRs for an system. Assume that the system has the following characteris- 
tics: 

¢ 96 MBytes of system memory is mapped as write-back memory (WB) for highest system performance. 

¢ Acustom 4-MByte I/O card is mapped to uncached memory (UC) at a base address of 64 MBytes. This 


restriction forces the 96 MBytes of system memory to be addressed from 0 to 64 MBytes and from 68 MBytes 
to 100 MBytes, leaving a 4-MByte hole for the I/O card. 


¢ An 8-MByte graphics card is mapped to write-combining memory (WC) beginning at address AOOQOOOOH. 
¢ The BIOS area from 15 MBytes to 16 MBytes is mapped to UC memory. 


The following settings for the MTRRs will yield the proper mapping of the physical address space for this system 
configuration. 


IA32_MTRR_PHYSBASEO = 0000 0000 0000 OO06H 
IA32_MTRR_PHYSMASKO = 0000 OOOF FCOO O800H 
Caches 0-64 MByte as WB cache type. 


IA32_MTRR_PHYSBASE1 = 0000 0000 0400 OOO6GH 
IA32_MTRR_PHYSMASK1 = 0000 OOOF FEOO O800H 
Caches 64-96 MByte as WB cache type. 


IA32_MTRR_PHYSBASE2 = 0000 0000 0600 OOO6H 
IA32_MTRR_PHYSMASK2 = 0000 OOOF FFCO O800H 
Caches 96-100 MByte as WB cache type. 


IA32_MTRR_PHYSBASE3 = 0000 0000 0400 OO00H 
IA32_MTRR_PHYSMASK3 = 0000 OOOF FFCO O800H 
Caches 64-68 MByte as UC cache type. 


IA32_MTRR_PHYSBASE4 = 0000 0000 OOFO OO00H 
IA32_MTRR_PHYSMASK4 = 0000 OOOF FFFO O800H 
Caches 15-16 MByte as UC cache type. 


IA32_MTRR_PHYSBASE5 = 0000 0000 A000 0001H 
IA32_MTRR_PHYSMASK5 = 0000 OOOF FF80 O800H 
Caches AOOOO000-A0800000 as WC type. 


This MTRR setup uses the ability to overlap any two memory ranges (as long as the ranges are mapped to WB and 
UC memory types) to minimize the number of MTRR registers that are required to configure the memory environ- 
ment. This setup also fulfills the requirement that two register pairs are left for operating system usage. 


11.11.3.1 Base and Mask Calculations for Greater-Than 36-bit Physical Address Support 


For Intel 64 and IA-32 processors that support greater than 36 bits of physical address size, software should query 
CPUID.80000008H to determine the maximum physical address. See the example. 


Example 11-3. Setting-Up Memory for a System with a 40-Bit Address Size 
If a processor supports 40-bits of physical address size, then the PhysMask field (in IA32_MTRR_PHYSMASKn 
registers) is 28 bits instead of 24 bits. For this situation, Example 11-2 should be modified as follows: 


IA32_MTRR_PHYSBASEO = 0000 0000 0000 OO06H 
IA32_MTRR_PHYSMASKO = 0000 OOFF FCOO 0800H 
Caches 0-64 MByte as WB cache type. 
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IA32_MTRR_PHYSBASE1 = 0000 0000 0400 OO06H 
IA32_MTRR_PHYSMASK1 = 0000 OOFF FE00 O800H 
Caches 64-96 MByte as WB cache type. 


IA32_MTRR_PHYSBASE2 = 0000 0000 0600 OO0O6H 
IA32_MTRR_PHYSMASK2 = 0000 OOFF FFCO O800H 
Caches 96-100 MByte as WB cache type. 


IA32_MTRR_PHYSBASE3 = 0000 0000 0400 OO00H 
IA32_MTRR_PHYSMASK3 = 0000 OOFF FFCO O800H 
Caches 64-68 MByte as UC cache type. 


IA32_MTRR_PHYSBASE4 = 0000 0000 OOFO OOOOH 
IA32_MTRR_PHYSMASK4 = 0000 OOFF FFFO O800H 
Caches 15-16 MByte as UC cache type. 


IA32_MTRR_PHYSBASE5 = 0000 0000 A000 0001H 
IA32_MTRR_PHYSMASK5 = 0000 OOFF FF80 O800H 
Caches AOOOO000-A0800000 as WC type. 


11.11.4 Range Size and Alignment Requirement 


A range that is to be mapped to a variable-range MTRR must meet the following “power of 2” size and alignment 
rules: 


1. The minimum range size is 4 KBytes and the base address of the range must be on at least a 4-KByte 
boundary. 


2. For ranges greater than 4 KBytes, each range must be of length 2” and its base address must be aligned ona 
2” boundary, where rn is a value equal to or greater than 12. The base-address alignment value cannot be less 
than its length. For example, an 8-KByte range cannot be aligned on a 4-KByte boundary. It must be aligned on 
at least an 8-KByte boundary. 


11.11.4.1 MTRR Precedences 


If the MTRRs are not enabled (by setting the E flag in the IA32_MTRR_DEF_TYPE MSR), then all memory accesses 
are of the UC memory type. If the MTRRs are enabled, then the memory type used for a memory access is deter- 
mined as follows: 


1. If the physical address falls within the first 1 MByte of physical memory and fixed MTRRs are enabled, the 
processor uses the memory type stored for the appropriate fixed-range MTRR. 


2. Otherwise, the processor attempts to match the physical address with a memory type set by the variable-range 
MTRRs: 


— If one variable memory range matches, the processor uses the memory type stored in the 
IA32_MTRR_PHYSBASEn register for that range. 


— Iftwoor more variable memory ranges match and the memory types are identical, then that memory type 
is used. 


— If two or more variable memory ranges match and one of the memory types is UC, the UC memory type 
used. 


— Iftwoor more variable memory ranges match and the memory types are WT and WB, the WT memory type 
is used. 


— For overlaps not defined by the above rules, processor behavior is undefined. 
3. If no fixed or variable memory range matches, the processor uses the default memory type. 
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11.11.5 MTRR Initialization 


On a hardware reset, the P6 and more recent processors clear the valid flags in variable-range MTRRs and clear the 
E flag in the IA32_MTRR_DEF_TYPE MSR to disable all MTRRs. All other bits in the MTRRs are undefined. 


Prior to initializing the MTRRs, software (normally the system BIOS) must initialize all fixed-range and variable- 
range MTRR register fields to 0. Software can then initialize the MTRRs according to known types of memory, 
including memory on devices that it auto-configures. Initialization is expected to occur prior to booting the oper- 
ating system. 


See Section 11.11.8, “MTRR Considerations in MP Systems,” for information on initializing MTRRs in MP (multiple- 
processor) systems. 


11.11.6 Remapping Memory Types 


A system designer may re-map memory types to tune performance or because a future processor may not imple- 
ment all memory types supported by the Pentium 4, Intel Xeon, and P6 family processors. The following rules 
support coherent memory-type re-mappings: 


1. A memory type should not be mapped into another memory type that has a weaker memory ordering model. 
For example, the uncacheable type cannot be mapped into any other type, and the write-back, write-through, 
and write-protected types cannot be mapped into the weakly ordered write-combining type. 


2. Amemory type that does not delay writes should not be mapped into a memory type that does delay writes, 
because applications of such a memory type may rely on its write-through behavior. Accordingly, the write- 
back type cannot be mapped into the write-through type. 


3. Amemory type that views write data as not necessarily stored and read back by a subsequent read, such as 
the write-protected type, can only be mapped to another type with the same behavior (and there are no others 
for the Pentium 4, Intel Xeon, and P6 family processors) or to the uncacheable type. 


In many specific cases, a system designer can have additional information about how a memory type is used, 
allowing additional mappings. For example, write-through memory with no associated write side effects can be 
mapped into write-back memory. 


11.11.7 | MTRR Maintenance Programming Interface 


The operating system maintains the MTRRs after booting and sets up or changes the memory types for memory- 
mapped devices. The operating system should provide a driver and application programming interface (API) to 
access and set the MTRRs. The function calls MemTypeGet() and MemTypeSet() define this interface. 


11.11.7.1  MemTypeGet() Function 


The MemTypeGet() function returns the memory type of the physical memory range specified by the parameters 
base and size. The base address is the starting physical address and the size is the number of bytes for the memory 
range. The function automatically aligns the base address and size to 4-KByte boundaries. Pseudocode for the 
MemTypeGet() function is given in Example 11-4. 
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Example 11-4. MemTypeGet() Pseudocode 
#define MIXED_TYPES -1__ /* 0 < MIXED_TYPES || MIXED_TYPES > 256 */ 


IF CPU_FEATURES.MTRR /* processor supports MTRRs */ 
THEN 

Align BASE and SIZE to 4-KByte boundary; 

IF (BASE + SIZE) wrap physical-address space 
THEN return INVALID; 

Fl; 

IF MTRRdefType.€ = 0 
THEN return UC; 

Fl; 

FirstType := Get4KMemType (BASE); 

/* Obtains memory type for first 4-KByte range. */ 

/* See Get4KMemType (4KByteRange) in Example 11-5. */ 

FOR each additional 4-KByte range specified in SIZE 
NextType := Get4KMemType (4KByteRange); 
IF NextType != FirstType 

THEN return Mixed_Types; 

Fl; 

ROF; 

return FirstType; 

ELSE return UNSUPPORTED; 
Fl; 


If the processor does not support MTRRs, the function returns UNSUPPORTED. If the MTRRs are not enabled, then 
the UC memory type is returned. If more than one memory type corresponds to the specified range, a status of 
MIXED_TYPES is returned. Otherwise, the memory type defined for the range (UC, WC, WT, WB, or WP) is 
returned. 


The pseudocode for the Get4KMemType() function in Example 11-5 obtains the memory type for a single 4-KByte 
range at a given physical address. The sample code determines whether an PHY_ADDRESS falls within a fixed 
range by comparing the address with the known fixed ranges: 0 to 7FFFFH (64-KByte regions), 80000H to BFFFFH 
(16-KByte regions), and COOOOH to FFFFFH (4-KByte regions). If an address falls within one of these ranges, the 
appropriate bits within one of its MTRRs determine the memory type. 


Example 11-5. Get4KMemType() Pseudocode 


IF IA32_ MTRRCAP.FIX AND MTRRdefType.FE /* fixed registers enabled */ 
THEN IF PHY ADDRESS is within a fixed range 
return [A32_MTRR_FIX.Type; 
Fl; 
FOR each variable-range MTRR in |A32_MTRRCAP.VCNT 
IF IA32_MTRR_PHYSMASK.V = 0 
THEN continue; 
Fl; 
IF (PHY_ADDRESS AND IA32_MTRR_PHYSMASK.Mask) = 
(IA32_MTRR_PHYSBASE.Base 
AND IA32_MTRR_PHYSMASK.Mask) 
THEN 
return [IA32_MTRR_PHYSBASE.Type; 
Fl; 
ROF; 
return MTRRdefType.Type; 
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11.11.7.2 MemTypeSet() Function 


The MemTypeSet() function in Example 11-6 sets a MTRR for the physical memory range specified by the parame- 
ters base and size to the type specified by type. The base address and size are multiples of 4 KBytes and the size 
is not O. 


Example 11-6. MemTypeSet Pseudocode 


IF CPU_FEATURES.MTRR (* processor supports MTRRs *) 

THEN 
IF BASE and SIZE are not 4-KByte aligned or size is 0 
THEN return INVALID; 
FI; 
IF (BASE + SIZE) wrap 4-GByte address space 
THEN return INVALID; 


FL; 
IF TYPE is invalid for Pentium 4, Intel Xeon, and P6é family 
processors 
THEN return UNSUPPORTED; 
FI; 
IF TYPE is WC and not supported 
THEN return UNSUPPORTED; 
FI; 
IF IA32 MTRRCAP.FIX is set AND range can be mapped using a 
fixed-range MTRR 
THEN 
pre_mtrr_change() ; 
update affected MTRR; 
post_mtrr_change() ; 


FI; 


ELSE (* try to map using a variable MTRR pair *) 
IF ITA32 MTRRCAP.VCNT = 0 
THEN return UNSUPPORTED; 
IF conflicts with current variable ranges 
THEN return RANGE OVERLAP; 
IF no MTRRs available 
THEN return VAR_NOT_ AVAILABLE; 


FI: 
+r 


IF BASE and SIZE do not meet the power of 2 requirements for 
variable MTRRs 
THEN return INVALID VAR REQUEST; 
FI; 
pre_mtrr_change() ; 
Update affected MTRRs; 
post_mtrr_change() ; 


FI; 


pre_mtrr_change () 
BEGIN 
disable interrupts; 
Save current value of CR4; 
disable and flush caches; 
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flush TLBs; 
disable MTRRs; 
IF multiprocessing 
THEN maintain consistency through IPIs; 

FI; 

END 

post_mtrr_change() 

BEGIN 
flush caches and TLBs; 
enable MTRRs; 
enable caches; 
restore value of CR4; 
enable interrupts; 

END 


The physical address to variable range mapping algorithm in the MemTypeSet function detects conflicts with 
current variable range registers by cycling through them and determining whether the physical address in question 
matches any of the current ranges. During this scan, the algorithm can detect whether any current variable ranges 
overlap and can be concatenated into a single range. 


The pre_mtrr_change() function disables interrupts prior to changing the MTRRs, to avoid executing code with a 
partially valid MTRR setup. The algorithm disables caching by setting the CD flag and clearing the NW flag in control 
register CRO. The caches are invalidated using the WBINVD instruction. The algorithm flushes all TLB entries either 
by clearing the page-global enable (PGE) flag in control register CR4 (if PGE was already set) or by updating control 
register CR3 (if PGE was already clear). Finally, it disables MTRRs by clearing the E flag in the 
IA32_MTRR_DEF_TYPE MSR. 


After the memory type is updated, the post_mtrr_change() function re-enables the MTRRs and again invalidates 
the caches and TLBs. This second invalidation is required because of the processor's aggressive prefetch of both 
instructions and data. The algorithm restores interrupts and re-enables caching by setting the CD flag. 


An operating system can batch multiple MTRR updates so that only a single pair of cache invalidations occur. 


11.11.8 MTRR Considerations in MP Systems 


In MP (multiple-processor) systems, the operating systems must maintain MTRR consistency between all the 
processors in the system. The Pentium 4, Intel Xeon, and P6 family processors provide no hardware support to 
maintain this consistency. In general, all processors must have the same MTRR values. 


This requirement implies that when the operating system initializes an MP system, it must load the MTRRs of the 
boot processor while the E flag in register MTRRdefType is 0. The operating system then directs other processors to 
load their MTRRs with the same memory map. After all the processors have loaded their MTRRs, the operating 
system signals them to enable their MTRRs. Barrier synchronization is used to prevent further memory accesses 
until all processors indicate that the MTRRs are enabled. This synchronization is likely to be a shoot-down style 
algorithm, with shared variables and interprocessor interrupts. 


Any change to the value of the MTRRs in an MP system requires the operating system to repeat the loading and 
enabling process to maintain consistency, using the following procedure: 


Broadcast to all processors to execute the following code sequence. 

Disable interrupts. 

Wait for all processors to reach this point. 

Enter the no-fill cache mode. (Set the CD flag in control register CRO to 1 and the NW flag to 0.) 


Flush all caches using the WBINVD instructions. Note on a processor that supports self-snooping, CPUID 
feature flag bit 27, this step is unnecessary. 


Oi eee Ol oN 


6. If the PGE flag is set in control register CR4, flush all TLBs by clearing that flag. 
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7. Ifthe PGE flag is clear in control register CR4, flush all TLBs by executing a MOV from control register CR3 to 
another register and then a MOV from that register back to CR3. 


8. Disable all range registers (by clearing the E flag in register MTRRdefType). If only variable ranges are being 
modified, software may clear the valid bits for the affected register pairs instead. 


9. Update the MTRRs. 


10. Enable all range registers (by setting the E flag in register MTRRdefType). If only variable-range registers were 
modified and their individual valid bits were cleared, then set the valid bits for the affected ranges instead. 


11. Flush all caches and all TLBs a second time. (The TLB flush is required for Pentium 4, Intel Xeon, and P6 family 
processors. Executing the WBINVD instruction is not needed when using Pentium 4, Intel Xeon, and P6 family 
processors, but it may be needed in future systems. ) 


12. Enter the normal cache mode to re-enable caching. (Set the CD and NW flags in control register CRO to 0.) 
13. Set PGE flag in control register CR4, if cleared in Step 6 (above). 

14. Wait for all processors to reach this point. 

15. Enable interrupts. 


11.11.9 Large Page Size Considerations 


The MTRRs provide memory typing for a limited number of regions that have a 4 KByte granularity (the same gran- 
ularity as 4-KByte pages). The memory type for a given page is cached in the processor’s TLBs. When using large 
pages (2 MBytes, 4 MBytes, or 1 GBytes), a single page-table entry covers multiple 4-KByte granules, each with a 
single memory type. Because the memory type for a large page is cached in the TLB, the processor can behave in 
an undefined manner if a large page is mapped to a region of memory that MTRRs have mapped with multiple 
memory types. 


Undefined behavior can be avoided by insuring that all MTRR memory-type ranges within a large page are of the 
same type. If a large page maps to a region of memory containing different MTRR-defined memory types, the PCD 
and PWT flags in the page-table entry should be set for the most conservative memory type for that range. For 
example, a large page used for memory mapped I/O and regular memory is mapped as UC memory. Alternatively, 
the operating system can map the region using multiple 4-KByte pages each with its own memory type. 


The requirement that all 4-KByte ranges in a large page are of the same memory type implies that large pages with 
different memory types may suffer a performance penalty, since they must be marked with the lowest common 
denominator memory type. The same consideration apply to 1 GByte pages, each of which may consist of multiple 
2-Mbyte ranges. 


The Pentium 4, Intel Xeon, and P6 family processors provide special support for the physical memory range from 0 
to 4 MBytes, which is potentially mapped by both the fixed and variable MTRRs. This support is invoked when a 
Pentium 4, Intel Xeon, or P6 family processor detects a large page overlapping the first 1 MByte of this memory 
range with a memory type that conflicts with the fixed MTRRs. Here, the processor maps the memory range as 
multiple 4-KByte pages within the TLB. This operation ensures correct behavior at the cost of performance. To 
avoid this performance penalty, operating-system software should reserve the large page option for regions of 
memory at addresses greater than or equal to 4 MBytes. 


11.12 PAGE ATTRIBUTE TABLE (PAT) 


The Page Attribute Table (PAT) extends the JA-32 architecture’s page-table format to allow memory types to be 
assigned to regions of physical memory based on linear address mappings. The PAT is a companion feature to the 
MTRRs; that is, the MTRRs allow mapping of memory types to regions of the physical address space, where the PAT 
allows mapping of memory types to pages within the linear address space. The MTRRs are useful for statically 
describing memory types for physical ranges, and are typically set up by the system BIOS. The PAT extends the 
functions of the PCD and PWT bits in page tables to allow all five of the memory types that can be assigned with the 
MTRRs (plus one additional memory type) to also be assigned dynamically to pages of the linear address space. 


The PAT was introduced to IA-32 architecture on the Pentium III processor. It is also available in the Pentium 4 and 
Intel Xeon processors. 
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11.12.1 Detecting Support for the PAT Feature 


An operating system or executive can detect the availability of the PAT by executing the CPUID instruction with a 
value of 1 in the EAX register. Support for the PAT is indicated by the PAT flag (bit 16 of the values returned to EDX 
register). If the PAT is supported, the operating system or executive can use the IA32_PAT MSR to program the PAT. 
When memory types have been assigned to entries in the PAT, software can then use of the PAT-index bit (PAT) in 
the page-table and page-directory entries along with the PCD and PWT bits to assign memory types from the PAT 
to individual pages. 

Note that there is no separate flag or control bit in any of the control registers that enables the PAT. The PAT is 


always enabled on all processors that support it, and the table lookup always occurs whenever paging is enabled, 
in all paging modes. 


11.12.2 IA32_PAT MSR 


The IA32_PAT MSR is located at MSR address 277H (see Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 
64 and IA-32 Architectures Software Developer’s Manual, Volume 4). Figure 11-9. shows the format of the 64-bit 
IA32_PAT MSR. 


The IA32_PAT MSR contains eight page attribute fields: PAO through PA7. The three low-order bits of each field are 
used to specify a memory type. The five high-order bits of each field are reserved, and must be set to all Os. Each 
of the eight page attribute fields can contain any of the memory type encodings specified in Table 11-10. 


31 27 26 24 23 19 18 16 15 11 10 8 7 3 2 0 
Reserved PA3 Reserved PA2 Reserved PA1 Reserved PAO 


63 59 58 56 55 51 50 48 47 43 42 40 39 35 34 32 
Reserved PA7 Reserved PAG Reserved PAS Reserved PA4 


Figure 11-9. IA32_PAT MSR 
Note that for the P6 family processors, the IA32_PAT MSR is named the PAT MSR. 


Table 11-10. Memory Types That Can Be Encoded With PAT 


Encoding Mnemonic 
OOH Uncacheable (UC) 
01H Write Combining (WC) 
02H Reserved* 
03H Reserved* 
04H Write Through (WT) 
05H Write Protected (WP) 
O6H Write Back (WB) 
07H Uncached (UC-) 
O8H - FFH Reserved* 


NOTE: 
* Using these encodings will result in a general-protection exception (#GP). 
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11.12.3 Selecting a Memory Type from the PAT 


To select a memory type for a page from the PAT, a 3-bit index made up of the PAT, PCD, and PWT bits must be 
encoded in the page-table or page-directory entry for the page. Table 11-11 shows the possible encodings of the 
PAT, PCD, and PWT bits and the PAT entry selected with each encoding. The PAT bit is bit 7 in page-table entries that 
point to 4-KByte pages and bit 12 in paging-structure entries that point to larger pages. The PCD and PWT bits are 
bits 4 and 3, respectively, in paging-structure entries that point to pages of any size. 


The PAT entry selected for a page is used in conjunction with the MTRR setting for the region of physical memory 
in which the page is mapped to determine the effective memory type for the page, as shown in Table 11-7. 


Table 11-11. Selection of PAT Entries with PAT, PCD, and PWT Flags 


PAT PCD PWT PAT Entry 
0 0 0 PATO 
0 0 1 PAT1 
0 1 0 PAT2 
0 1 1 PAT3 
1 0 0 PAT4 
1 0 1 PAT5 
1 1 0 PAT6 
1 1 1 PAT7 


11.12.4 Programming the PAT 


Table 11-12 shows the default setting for each PAT entry following a power up or reset of the processor. The setting 
remain unchanged following a soft reset (INIT reset). 


Table 11-12. Memory Type Setting of PAT Entries Following a Power-up or Reset 


PAT Entry Memory Type Following Power-up or Reset 
PATO WB 
PAT1 WT 
PAT2 UC- 
PAT3 UC 
PAT4 WB 
PATS WT 
PAT6 UC- 
PAT7 UC 


The values in all the entries of the PAT can be changed by writing to the IA32_PAT MSR using the WRMSR instruc- 
tion. The IA32_PAT MSR is read and write accessible (use of the RDMSR and WRMSR instructions, respectively) to 
software operating at a CPL of 0. Table 11-10 shows the allowable encoding of the entries in the PAT. Attempting to 
write an undefined memory type encoding into the PAT causes a general-protection (#GP) exception to be gener- 
ated. 


The operating system is responsible for insuring that changes to a PAT entry occur in a manner that maintains the 
consistency of the processor caches and translation lookaside buffers (TLB). This is accomplished by following the 
procedure as specified in Section 11.11.8, “MTRR Considerations in MP Systems,” for changing the value of an 
MTRR in a multiple processor system. It requires a specific sequence of operations that includes flushing the 
processors caches and TLBs. 


The PAT allows any memory type to be specified in the page tables, and therefore it is possible to have a single 
physical page mapped to two or more different linear addresses, each with different memory types. Intel does not 
support this practice because it may lead to undefined operations that can result in a system failure. In particular, 
a WC page must never be aliased to a cacheable page because WC writes may not check the processor caches. 
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When remapping a page that was previously mapped as a cacheable memory type to a WC page, an operating 
system can avoid this type of aliasing by doing the following: 


1. Remove the previous mapping to a cacheable memory type in the page tables; that is, make them not 
present. 


Flush the TLBs of processors that may have used the mapping, even speculatively. 
Create a new mapping to the same physical address with a new memory type, for instance, WC. 


Flush the caches on all processors that may have used the mapping previously. Note on processors that support 
self-snooping, CPUID feature flag bit 27, this step is unnecessary. 


Operating systems that use a page directory as a page table (to map large pages) and enable page size extensions 
must carefully scrutinize the use of the PAT index bit for the 4-KByte page-table entries. The PAT index bit for a 
page-table entry (bit 7) corresponds to the page size bit in a page-directory entry. Therefore, the operating system 
can only use PAT entries PAO through PA3 when setting the caching type for a page table that is also used as a page 
directory. If the operating system attempts to use PAT entries PA4 through PA7 when using this memory as a page 
table, it effectively sets the PS bit for the access to this memory as a page directory. 


For compatibility with earlier IA-32 processors that do not support the PAT, care should be taken in selecting the 
encodings for entries in the PAT (see Section 11.12.5, “PAT Compatibility with Earlier IA-32 Processors”). 


11.12.5 PAT Compatibility with Earlier |A-32 Processors 


For IA-32 processors that support the PAT, the IA32_PAT MSR is always active. That is, the PCD and PWT bits in 
page-table entries and in page-directory entries (that point to pages) are always select a memory type for a page 
indirectly by selecting an entry in the PAT. They never select the memory type for a page directly as they do in 
earlier IA-32 processors that do not implement the PAT (see Table 11-6). 


To allow compatibility for code written to run on earlier IA-32 processor that do not support the PAT, the PAT mech- 
anism has been designed to allow backward compatibility to earlier processors. This compatibility is provided 
through the ordering of the PAT, PCD, and PWT bits in the 3-bit PAT entry index. For processors that do not imple- 
ment the PAT, the PAT index bit (bit 7 in the page-table entries and bit 12 in the page-directory entries) is reserved 
and set to 0. With the PAT bit reserved, only the first four entries of the PAT can be selected with the PCD and PWT 
bits. At power-up or reset (see Table 11-12), these first four entries are encoded to select the same memory types 
as the PCD and PWT bits would normally select directly in an IA-32 processor that does not implement the PAT. So, 
if encodings of the first four entries in the PAT are left unchanged following a power-up or reset, code written to run 
on earlier IA-32 processors that do not implement the PAT will run correctly on IA-32 processors that do implement 
the PAT. 
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CHAPTER 12 
INTEL® MMX™ TECHNOLOGY SYSTEM PROGRAMMING 


This chapter describes those features of the Intel® MMX™ technology that must be considered when designing or 
enhancing an operating system to support MMX technology. It covers MMX instruction set emulation, the MMX 
state, aliasing of MMX registers, saving MMX state, task and context switching considerations, exception handling, 


and debugging. 


12.1 EMULATION OF THE MMX INSTRUCTION SET 


The IA-32 or Intel 64 architecture does not support emulation of the MMX instructions, as it does for x87 FPU 
instructions. The EM flag in control register CRO (provided to invoke emulation of x87 FPU instructions) cannot be 
used for MMX instruction emulation. If an MMX instruction is executed when the EM flag is set, an invalid opcode 
exception (UD#) is generated. Table 12-1 shows the interaction of the EM, MP, and TS flags in control register CRO 


when executing MMX instructions. 


Table 12-1. Action Taken By MMX Instructions for Different Combinations of EM, MP and TS 


CRO Flags 
EM MP* TS Action 
0 1 0 Execute. 
0 1 1 #NM exception. 
1 1 0 #UD exception. 
1 1 1 #UD exception. 


NOTE: 
* For processors that support the MMX instructions, the MP flag should be set. 


12.2 THE MMX STATE AND MMX REGISTER ALIASING 


The MMxX state consists of eight 64-bit registers (MMO through MM7). These registers are aliased to the low 64-bits 
(bits 0 through 63) of floating-point registers RO through R7 (see Figure 12-1). Note that the MMX registers are 
mapped to the physical locations of the floating-point registers (RO through R7), not to the relative locations of the 
registers in the floating-point register stack (STO through ST7). As a result, the MMX register mapping is fixed and 
is not affected by value in the Top Of Stack (TOS) field in the floating-point status word (bits 11 through 13). 
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Figure 12-1. Mapping of MMX Registers to Floating-Point Registers 


When a value is written into an MMX register using an MMX instruction, the value also appears in the corresponding 
floating-point register in bits 0 through 63. Likewise, when a floating-point value written into a floating-point 
register by a x87 FPU, the low 64 bits of that value also appears in a the corresponding MMxX register. 

The execution of MMX instructions have several side effects on the x87 FPU state contained in the floating-point 
registers, the x87 FPU tag word, and the x87 FPU status word. These side effects are as follows: 

When an MM<x instruction writes a value into an MMX register, at the same time, bits 64 through 79 of the corre- 
sponding floating-point register are set to all 1s. 


When an MMx instruction (other than the EMMS instruction) is executed, each of the tag fields in the x87 FPU 
tag word is set to 0OB (valid). (See also Section 12.2.1, “Effect of MMX, x87 FPU, FXSAVE, and FXRSTOR 


Instructions on the x87 FPU Tag Word.”) 
When the EMMS instruction is executed, each tag field in the x87 FPU tag word is set to 11B (empty). 


e 
¢ Each time an MMxX instruction is executed, the TOS value is set to OOOB. 


Execution of MMX instructions does not affect the other bits in the x87 FPU status word (bits 0 through 10 and bits 
14 and 15) or the contents of the other x87 FPU registers that comprise the x87 FPU state (the x87 FPU control 


word, instruction pointer, data pointer, or opcode registers). 
Table 12-2 summarizes the effects of the MMX instructions on the x87 FPU state. 
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Table 12-2. Effects of MMX Instructions on x87 FPU State 


MMxX Instruction x87 FPU Tag TOS Field of x87 | Other x87 FPU Bits 64 Through 79 ‘| Bits O Through 63 

Type Word FPU Status Word | Registers of x87 FPU Data of x87 FPU Data 
Registers Registers 

Read from MMX All tags set to OOB | OOOB Unchanged Unchanged Unchanged 

register (Valid) 

Write to MMX All tags set to 0OB | OOOB Unchanged Set to all 1s Overwritten with 

register (Valid) MMX data 

EMMS All fields set to OOOB Unchanged Unchanged Unchanged 

11B (Empty) 
12.2.1 Effect of MMX, x87 FPU, FXSAVE, and FXRSTOR 


Instructions on the x87 FPU Tag Word 


Table 12-3 summarizes the effect of MMX and x87 FPU instructions and the FXSAVE and FXRSTOR instructions on 
the tags in the x87 FPU tag word and the corresponding tags in an image of the tag word stored in memory. 


The values in the fields of the x87 FPU tag word do not affect the contents of the MMX registers or the execution of 
MMX instructions. However, the MMX instructions do modify the contents of the x87 FPU tag word, as is described 
in Section 12.2, “The MMX State and MMX Register Aliasing.” These modifications may affect the operation of the 
x87 FPU when executing x87 FPU instructions, if the x87 FPU state is not initialized or restored prior to beginning 
x87 FPU instruction execution. 


Note that the FSAVE, FXSAVE, and FSTENV instructions (which save x87 FPU state information) read the x87 FPU 
tag register and contents of each of the floating-point registers, determine the actual tag values for each register 
(empty, nonzero, zero, or special), and store the updated tag word in memory. After executing these instructions, 
all the tags in the x87 FPU tag word are set to empty (11B). Likewise, the EMMS instruction clears MMX state from 
the MMX/floating-point registers by setting all the tags in the x87 FPU tag word to 11B. 


Table 12-3. Effect of the MMX, x87 FPU, and FXSAVE/FXRSTOR Instructions on the x87 FPU Tag Word 
Instruction Instruction x87 FPU Tag Word Image of x87 FPU Tag Word Stored in 
Type Memory 
MMX All (except EMMS) All tags are set to OOB (valid). Not affected. 

MMX EMMS All tags are set to 11B (empty). Not affected. 
x87 FPU All (except FSAVE, Tag for modified floating-point register is | Not affected. 
FSTENV, FRSTOR, set to OOB or 11B. 
FLDENV) 
x87 FPU and FSAVE, FSTENV, FXSAVE | Tags and register values are read and Tags are set according to the actual 
FXSAVE interpreted; then all tags are set to 11B. | values in the floating-point registers; 
that is, empty registers are marked 11B 
and valid registers are marked OOB 
(nonzero), 01B (zero), or 10B (special). 
x87 FPU and FRSTOR, FLDENV, All tags marked 11B in memory are set | Tags are read and interpreted, but not 
FXRSTOR FXRSTOR to 11B; all other tags are set according modified. 
to the value in the corresponding 
floating-point register: OOB (nonzero), 
01B (zero), or 10B (special). 
12.3. SAVING AND RESTORING THE MMX STATE AND REGISTERS 


Because the MMX registers are aliased to the x87 FPU data registers, the MMX state can be saved to memory and 
restored from memory as follows: 
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° Execute an FSAVE, FNSAVE, or FXSAVE instruction to save the MMX state to memory. (The FXSAVE instruction 
also saves the state of the XMM and MXCSR registers. ) 


¢ Execute an FRSTOR or FXRSTOR instruction to restore the MMX state from memory. (The FXRSTOR instruction 
also restores the state of the XMM and MXCSR registers. ) 


The save and restore methods described above are required for operating systems (see Section 12.4, “Saving MMX 
State on Task or Context Switches”). Applications can in some cases save and restore only the MMX registers in the 
following way: 


° Execute eight MOVQ instructions to save the contents of the MMXO through MMX7 registers to memory. An 
EMMS instruction may then (optionally) be executed to clear the MMX state in the x87 FPU. 


° Execute eight MOVQ instructions to read the saved contents of MMX registers from memory into the MMXO 
through MMX7 registers. 


NOTE 


The IA-32 architecture does not support scanning the x87 FPU tag word and then only saving valid 
entries. 


12.4 SAVING MMX STATE ON TASK OR CONTEXT SWITCHES 


When switching from one task or context to another, it is often necessary to save the MMX state. As a general rule, 
if the existing task switching code for an operating system includes facilities for saving the state of the x87 FPU, 
these facilities can also be relied upon to save the MMX state, without rewriting the task switch code. This reliance 
is possible because the MMX state is aliased to the x87 FPU state (see Section 12.2, “The MMX State and MMX 
Register Aliasing”). 


With the introduction of the FXSAVE and FXRSTOR instructions and of SSE/SSE2/SSE3/SSSE3 extensions, it is 
possible (and more efficient) to create state saving facilities in the operating system or executive that save the x87 
FPU/MMX/SSE/SSE2/SSE3/SSSE3 state in one operation. Section 13.4, “Designing OS Facilities for Saving x87 
FPU, SSE AND EXTENDED States on Task or Context Switches,” describes how to design such facilities. The tech- 
niques describes in this section can be adapted to saving only the MMX and x87 FPU state if needed. 


12.5 EXCEPTIONS THAT CAN OCCUR WHEN EXECUTING MMX INSTRUCTIONS 


MMX instructions do not generate x87 FPU floating-point exceptions, nor do they affect the processor's status flags 
in the EFLAGS register or the x87 FPU status word. The following exceptions can be generated during the execution 
of an MMX instruction: 


° Exceptions during memory accesses: 

— Stack-segment fault (#SS). 

— General protection (#GP). 

— Page fault (#PF). 

— Alignment check (#AC), if alignment checking is enabled. 
¢ System exceptions: 


— Invalid Opcode (#UD), if the EM flag in control register CRO is set when an MMxX instruction is executed (see 
Section 12.1, “Emulation of the MMX Instruction Set”). 


— Device not available (#NM), if an MMX instruction is executed when the TS flag in control register CRO is 
set. (See Section 13.4.1, “Using the TS Flag to Control the Saving of the x87 FPU and SSE State.”) 


¢ Floating-point error (#MF). (See Section 12.5.1, “Effect of MMX Instructions on Pending x87 Floating-Point 
Exceptions.”) 


* Other exceptions can occur indirectly due to the faulty execution of the exception handlers for the above 
exceptions. 


12-4 Vol. 3A 


INTEL® MMX™ TECHNOLOGY SYSTEM PROGRAMMING 


12.5.1 €ffect of MMX Instructions on Pending x87 Floating-Point Exceptions 


If an x87 FPU floating-point exception is pending and the processor encounters an MMX instruction, the processor 
generates a x87 FPU floating-point error (#MF) prior to executing the MMxX instruction, to allow the pending excep- 
tion to be handled by the x87 FPU floating-point error exception handler. While this exception handler is executing, 
the x87 FPU state is maintained and is visible to the handler. Upon returning from the exception handler, the MMX 
instruction is executed, which will alter the x87 FPU state, as described in Section 12.2, “The MMX State and MMX 
Register Aliasing.” 


12.6 DEBUGGING MMX CODE 


The debug facilities operate in the same manner when executing MMX instructions as when executing other IA-32 
or Intel 64 architecture instructions. 


To correctly interpret the contents of the MMX or x87 FPU registers from the FSAVE/FNSAVE or FXSAVE image in 
memory, a debugger needs to take account of the relationship between the x87 FPU register’s logical locations 
relative to TOS and the MMX register’s physical locations. 


In the x87 FPU context, STn refers to an x87 FPU register at location n relative to the TOS. However, the tags in the 
x87 FPU tag word are associated with the physical locations of the x87 FPU registers (RO through R7). The MMX 
registers always refer to the physical locations of the registers (with MMO through MM7 being mapped to RO 
through R7). Figure 12-2 shows this relationship. Here, the inner circle refers to the physical location of the x87 
FPU and MMX registers. The outer circle refers to the x87 FPU registers’s relative location to the current TOS. 


When the TOS equals 0 (case A in Figure 12-2), STO points to the physical location RO on the floating-point stack. 
MMO maps to STO, MM1 maps to ST1, and so on. 


When the TOS equals 2 (case B in Figure 12-2), STO points to the physical location R2. MMO maps to ST6, MM1 
maps to ST7, MM2 maps to STO, and so on. 


beasts a, 87 FPU “push” 
x87 FPU push’ <7 x87 FPU “pop ste * i 


a 


Case A: TOS=0 Case B: TOS=2 


Outer circle = x87 FPU data register’s logical location relative to TOS 
Inner circle = x87 FPU tags = MMX register’s location = FP registers’s physical location 


Figure 12-2. Mapping of MMX Registers to x87 FPU Data Register Stack 
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CHAPTER 13 
SYSTEM PROGRAMMING FOR INSTRUCTION SET EXTENSIONS AND 
PROCESSOR EXTENDED STATES 


This chapter describes system programming features for instruction set extensions operating on the processor 
state extension known as the SSE state (XMM registers, MXCSR) and for other processor extended states. Instruc- 
tion set extensions operating on the SSE state include the streaming SIMD extensions (SSE), streaming SIMD 
extensions 2 (SSE2), streaming SIMD extensions 3 (SSE3), Supplemental SSE3 (SSSE3), and SSE4. Collectively, 
these are called SSE extensions! and the corresponding instructions SSE instructions. FXSAVE/FXRSTOR 
instructions can be used save/restore SSE state along with FP state. See Section 10.5 in the Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 1 for information about FXSAVE and FXRSTOR. 


Sections 13.1 through 13.4 cover system programming requirements to enable the SSE extensions, providing 
operating system or executive support for the SSE extensions, SIMD floating-point exceptions, exception handling, 
and task (context) switching. These sections primarily discuss use of FXSAVE/FXRSTOR to save/restore SSE state. 


XSAVE feature set refers to extensions to the Intel architecture that will allow system executives to implement 
support for multiple processor extended states along with FP/SSE states that may be introduced over time without 
requiring the system executive to be modified each time a new processor state extension is introduced. XSAVE 
feature set provide mechanisms to enumerate the supported extended states, enable some or all of them for soft- 
ware use, instructions to save/restore the states and enumerate the layout of the states when saved to memory. 
XSAVE/XRSTOR instructions are part of the XSAVE feature set. These instructions are introduced after the intro- 
duction of FP/SSE states but can be used to manage legacy FP/SSE state along with processor extended states. See 
CHAPTER 13 in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 for information 
about XSAVE feature set. 


System programming for managing processor extended states is described in sections 13.5 through 13.6. XSAVE 
feature set is designed to be compatible with FXSAVE/FXRSTOR and hence much of the material through sections 
13.1 to 13.4 related to SSE state also applies to XSAVE feature set with the exception of enumeration and 
saving/restoring state. 


XSAVE Compaction is an XSAVE feature that allows operating systems to allocate space for only the states saved 
to conserve memory usage. A new instruction called XSAVEC is introduced to save extended states in compacted 
format and XRSTOR instruction is enhanced to comprehend compacted format. System programming for managing 
processor extended states in compacted format is also described in section 13.5. 


Supervisor state is an extended state that can only be accessed in ring 0. XSAVE feature set has been enhanced 
to manage supervisor states. Two new ring 0 instructions, XSAVES/XRSTORS, are introduced to save/restore 
supervisor states along with other XSAVE managed states. They are privileged instruction and only operate in 
compacted format. System programming for managing supervisor states in described in section 13.7. 


Each XSAVE managed features may have additional feature specific system programming requirements such as 
exception handlers etc. Feature specific system programming requirements for XSAVE managed features are 
described in section 13.8. 


13.1 PROVIDING OPERATING SYSTEM SUPPORT FOR SSE EXTENSIONS 


To use SSE extensions, the operating system or executive must provide support for initializing the processor to use 
these extensions, for handling SIMD floating-point exceptions, and for using FXKSAVE and FXRSTOR (Section 10.5 
of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1) to manage context. XSAVE 
feature set can also be used to manage SSE state along with other processor extended states as described in 13.5. 
This section primarily focuses on using FXSAVE/FXRSTOR to manage SSE state. Because SSE extensions share the 
same state, experience the same sets of non-numerical and numerical exception behavior, these guidelines that 
apply to SSE also apply to other sets of SIMD extensions that operate on the same processor state and subject to 
the same sets of non-numerical and numerical exception behavior. 


1. The collection also includes PCLMULQDQ and AES instructions operating on XMM state. 
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Chapter 11, “Programming with Streaming SIMD Extensions 2 (SSE2)” and Chapter 12, “Programming with Intel® 
SSE3, SSSE3,Intel® SSE4 AND Intel® AESNI,” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 1, provide details on SSE instruction set. 


13.1.1. Adding Support to an Operating System for SSE Extensions 


The following guidelines describe functions that an operating system or executive must perform to support SSE 
extensions: 


1. Check that the processor supports the SSE extensions. 

Check that the processor supports the FXSAVE and FXRSTOR instructions or the XSAVE feature set. 
Provide an initialization for the SSE states. 

Provide support for the FXSAVE and FXRSTOR instructions or the XSAVE feature set. 


Provide support (if necessary) in non-numeric exception handlers for exceptions generated by the SSE instruc- 
tions. 


G1) ee 


6. Provide an exception handler for the SIMD floating-point exception (#XM). 
The following sections describe how to implement each of these guidelines. 


13.1.2 Checking for CPU Support 


If the processor attempts to execute an unsupported SSE instruction, the processor generates an invalid-opcode 
exception (#UD). Before an operating system or executive attempts to use SSE extensions, it should check that 
support is present by confirming the following bit values returned by the CPUID instruction: 


® CPUID.1:EDX.SSE[bit 25] = 1 

® CPUID.1:EDX.SSE2[bit 26] = 1 

® CPUID.1:ECX.SSE3[bit 0] = 1 

® CPUID.1:ECX.SSSE3[bit 9] = 1 

® CPUID.1:ECX.SSE4_1[bit 19] = 1 

® CPUID.1:ECX.SSE4_2[bit 20] = 1 

(To use POPCNT instruction, software must check CPUID.1:ECX.POPCNT[bit 23] = 1.) 


Separate checks must be made to ensure that the processor supports either FXSAVE and FXRSTOR or the XSAVE 
feature set. See Section 10.5 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 and 
Chapter 13 of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, respectively. 


13.1.3 Initialization of the SSE Extensions 


The operating system or executive should carry out the following steps to set up SSE extensions for use by appli- 
cation programs: 


1. Set CR4.0SFXSR[bit 9] = 1. Setting this flag implies that the operating system provides facilities for saving 
and restoring SSE state using FXSAVE and FXRSTOR instructions. These instructions may be used to save the 
SSE state during task switches and when invoking the SIMD floating-point exception (#XM) handler (see 
Section 13.1.5, “Providing a Handler for the SIMD Floating-Point Exception (#XM)”). 


If the processor does not support the FXSAVE and FXRSTOR instructions, attempting to set the OSFXSR flag 
causes a general-protection exception (#GP) to be generated. 
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2. Set CR4.0SXMMEXCPT[bit 10] = 1. Setting this flag implies that the operating system provides a SIMD 
floating-point exception (#XM) handler (see Section 13.1.5, “Providing a Handler for the SIMD Floating-Point 
Exception (#XM)”"). 


NOTE 


The OSFXSR and OSXMMEXCPT bits in control register CR4 must be set by the operating system. 
The processor has no other way of detecting operating-system support for the FXSAVE and 
FXRSTOR instructions or for handling SIMD floating-point exceptions. 


3. Clear CRO.EM[bit 2] = 0. This action disables emulation of the x87 FPU, which is required when executing SSE 


instructions (see Section 2.5, “Control Registers”). 


4. Set CRO.MP[bit 1] = 1. This setting is required for Intel 64 and IA-32 processors that support the SSE 
extensions (see Section 9.2.1, “Configuring the x87 FPU Environment”). 


Table 13-1 and Table 13-2 show the actions of the processor when an SSE instruction is executed, depending on 


the following: 


¢ OSFXSR and OSXMMEXCPT flags in control register CR4 
° SSE/SSE2/SSE3/SSSE3/SSE4 feature flags returned by CPUID 


° EM, MP, and TS flags in control register CRO 


Table 13-1. Action Taken for Combinations of OSFXSR, OSXMMEXCPT, SSE, SSE2, SSE3, EM, MP, and TS! 


CR4 CPUID CRO Flags 
OSFXSR OSXMMEXCPT SSE, SSE2, EM Mp4 TS Action 
SSE32, 
sse4_13 
0 x° X X 1 X | #UD exception. 
1 0 X 1 X | #UD exception. 
1 1 1 1 X | #UD exception. 
1 0 1 0 1 QO | Execute instruction; HUD exception if unmasked 
SIMD floating-point exception is detected. 
1 1 1 0 1 QO | Execute instruction; #XM exception if unmasked 
SIMD floating-point exception is detected. 
1 X 1 0 1 1 #NM exception. 
NOTES: 


1. For execution of any SSE instruction except the PAUSE, PREFETCHh, SFENCE, LFENCE, MFENCE, MOVNTI, and CLFLUSH instructions. 
2. Exception conditions due to CR4.0SFXSR or CR4.0SXMMEXCPT do not apply to FISTTP. 


3. Only applies to DPPS, DPPD, ROUNDPS, ROUNDPD, ROUNDSS, ROUNDSD. 


4. For processors that support the MMX instructions, the MP flag should be set. 


5. X = Don't care. 
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Table 13-2. Action Taken for Combinations of OSFXSR, SSSE3, SSE4, EM, and TS 


CR4 CPUID CRO Flags 
OSFXSR SSSE3 EM TS Action 

SSE4_1! 
SSE4_27 

0 x3 x X #UD exception. 

1 0 X X #UD exception. 

1 1 1 X #UD exception. 

1 1 0 1 #NM exception. 


NOTES: 

1. Applies to SSE4_1 instructions except DPPS, DPPD, ROUNDPS, ROUNDPD, ROUNDSS, ROUNDSD. 
2. Applies to SSE4_2 instructions except CRC32 and POPCNT. 

3. X = Don't care. 


The SIMD floating-point exception mask bits (bits 7 through 12), the flush-to-zero flag (bit 15), the denormals-are- 
zero flag (bit 6), and the rounding control field (bits 13 and 14) in the MXCSR register should be left in their default 
values of 0. This permits the application to determine how these features are to be used. 


13.1.4 Providing Non-Numeric Exception Handlers for Exceptions Generated by the SSE 
Instructions 

SSE instructions can generate the same type of memory-access exceptions (such as page faults and limit viola- 

tions) and other non-numeric exceptions as other Intel 64 and IA-32 architecture instructions generate. 


Ordinarily, existing exception handlers can handle these and other non-numeric exceptions without code modifica- 
tion. However, depending on the mechanisms used in existing exception handlers, some modifications might need 
to be made. 


The SSE extensions can generate the non-numeric exceptions listed below: 
¢ Memory Access Exceptions: 
— Stack-segment fault (#SS). 


— General protection exception (#GP). Executing most SSE instructions with an unaligned 128-bit memory 
reference generates a general-protection exception. (The MOVUPS and MOVUPD instructions allow 
unaligned a loads or stores of 128-bit memory locations, without generating a general-protection 
exception.) A 128-bit reference within the stack segment that is not aligned to a 16-byte boundary will also 
generate a general-protection exception, instead a stack-segment fault exception (#SS). 


— Page fault (#PF). 


— Alignment check (#AC). When enabled, this type of alignment check operates on operands that are less 
than 128-bits in size: 16-bit, 32-bit, and 64-bit. To enable the generation of alignment check exceptions, do 
the following: 


¢ Set the AM flag (bit 18 of control register CRO) 
* Set the AC flag (bit 18 of the EFLAGS register) 
* CPL must be 3 


If alignment check exceptions are enabled, 16-bit, 32-bit, and 64-bit misalignment will be detected for the 
MOVUPD and MOVUPS instructions; detection of 128-bit misalignment is not guaranteed and may vary 
with implementation. 
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° System Exceptions: 


— Invalid-opcode exception (#UD). This exception is generated when executing SSE instructions under the 
following conditions: 


¢ SSE/SSE2/SSE3/SSSE3/SSE4_1/SSE4_2 feature flags returned by CPUID are set to 0. This condition 
does not affect the CLFLUSH instruction, nor POPCNT. 


¢ The CLFSH feature flag returned by the CPUID instruction is set to 0. This exception condition only 
pertains to the execution of the CLFLUSH instruction. 


¢ The POPCNT feature flag returned by the CPUID instruction is set to 0. This exception condition only 
pertains to the execution of the POPCNT instruction. 


¢ The EM flag (bit 2) in control register CRO is set to 1, regardless of the value of TS flag (bit 3) of CRO. 
This condition does not affect the PAUSE, PREFETCHA, MOVNTI, SFENCE, LFENCE, MFENCE, CLFLUSH, 
CRC32 and POPCNT instructions. 


¢ The OSFXSR flag (bit 9) in control register CR4 is set to 0. This condition does not affect the PSHUFW, 
MOVNTQ, MOVNTI, PAUSE, PREFETCHA, SFENCE, LFENCE, MFENCE, CLFLUSH, CRC32 and POPCNT 
instructions. 


¢ Executing an instruction that causes a SIMD floating-point exception when the OSXMMEXCPT flag (bit 
10) in control register CR4 is set to 0. See Section 13.4.1, “Using the TS Flag to Control the Saving of 
the x87 FPU and SSE State.” 


— Device not available (#NM). This exception is generated by executing a SSE instruction when the TS flag 
(bit 3) of CRO is set to 1. 


Other exceptions can occur during delivery of the above exceptions. 


13.1.5 Providing a Handler for the SIMD Floating-Point Exception (#XM) 


SSE instructions do not generate numeric exceptions on packed integer operations. They can generate the 
following numeric (SIMD floating-point) exceptions on packed and scalar single-precision and double-precision 
floating-point operations. 


¢ Invalid operation (#1) 

* Divide-by-zero (#Z) 

¢ Denormal operand (#D) 

¢ Numeric overflow (#0) 

¢ Numeric underflow (#U) 

¢  Inexact result (Precision) (#P) 


These SIMD floating-point exceptions (with the exception of the denormal operand exception) are defined in the 
IEEE Standard 754 for Binary Floating-Point Arithmetic and represent the same conditions that cause x87 FPU 
floating-point error exceptions (#MF) to be generated for x87 FPU instructions. 


Each of these exceptions can be masked, in which case the processor returns a reasonable result to the destination 
operand without invoking an exception handler. However, if any of these exceptions are left unmasked, detection 
of the exception condition results in a SIMD floating-point exception (#XM) being generated. See Chapter 6, 
“Interrupt 19—SIMD Floating-Point Exception (#XM).” 


To handle unmasked SIMD floating-point exceptions, the operating system or executive must provide an exception 
handler. The section titled “SSE and SSE2 SIMD Floating-Point Exceptions” in Chapter 11, “Programming with 
Streaming SIMD Extensions 2 (SSE2),” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 1, describes the SIMD floating-point exception classes and gives suggestions for writing an exception 
handler to handle them. 


To indicate that the operating system provides a handler for SIMD floating-point exceptions (#XM), the OSXM- 
MEXCPT flag (bit 10) must be set in control register CR4. 
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13.1.5.1 Numeric Error flag and IGNNE# 


SSE extensions ignore the NE flag in control register CRO (that is, they treat it as if it were always set) and the 
IGNNE# pin. When an unmasked SIMD floating-point exception is detected, it is always reported by generating a 
SIMD floating-point exception (#XM). 


13.2 EMULATION OF SSE EXTENSIONS 


The Intel 64 and IA-32 architectures do not support emulation of the SSE instructions, as they do for x87 FPU 
instructions. 


The EM flag in control register CRO (provided to invoke emulation of x87 FPU instructions) cannot be used to invoke 
emulation of SSE instructions. If an SSE instruction is executed when CRO.EM = 1, an invalid opcode exception 
(#UD) is generated. See Table 13-1. 


13.3. SAVING AND RESTORING SSE STATE 


The SSE state consists of the state of the XMM and MXCSR registers. Intel recommends the following method for 
saving and restoring this state: 


¢ Execute the FXSAVE instruction to save the state of the XMM and MXCSR registers to memory. 


° Execute the FXRSTOR instruction to restore the state of the XMM and MXCSR registers from the image saved in 
memory earlier. 


This save and restore method is required for all operating systems. XSAVE feature set can also be used to 
save/restore SSE state. See Section 13.5, “The XSAVE Feature Set and Processor Extended State Management,” 
for using the XSAVE feature set to save/restore SSE state. 


In some cases, applications may choose to save only the XMM and MXCSR registers in the following manner: 
¢ Execute MOVDQ instructions to save the contents of the XMM registers to memory. 

¢ Execute a STMXCSR instruction to save the state of the MXCSR register to memory. 

Such applications must restore the XMM and MXCSR registers as follows: 


¢ Execute MOVDQ instructions to load the saved contents of the XMM registers from memory into the XMM 
registers. 


¢ Execute a LDMXCSR instruction to restore the state of the MXCSR register from memory. 


13.4 DESIGNING OS FACILITIES FOR SAVING X87 FPU, SSE AND EXTENDED 
STATES ON TASK OR CONTEXT SWITCHES 


The x87 FPU and SSE state consist of the state of the x87 FPU, XMM, and MXCSR registers. The FXSAVE and 
FXRSTOR instructions provide a fast method for saving and restoring this state. The XSAVE feature set can also be 
used to save FP and SSE state along with other extended states (see Section 13.5). 


Older operating systems may use FSAVE/FNSAVE and FRSTOR to save the x87 FPU state. These facilities can be 
extended to save and restore SSE state by substituting FXSAVE and FXRSTOR or the XSAVE feature set in place of 
FSAVE/FNSAVE and FRSTOR. 


If task or context switching facilities are written from scratch, any of several approaches may be taken for using the 
FXSAVE and FXRSTOR instructions or the XSAVE feature set to save and restore x87 FPU and SSE state: 


¢ The operating system can require applications that are intended to be run as tasks take responsibility for saving 
the states prior to a task suspension during a task switch and for restoring the states when the task is resumed. 
This approach is appropriate for cooperative multitasking operating systems, where the application has control 
over (or is able to determine) when a task switch is about to occur and can save state prior to the task switch. 
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¢ The operating system can take the responsibility for saving the states as part of the task switch process and 
restoring the state of the registers when a suspended task is resumed. This approach is appropriate for 
preemptive multitasking operating systems, where the application cannot know when it is going to be 
preempted and cannot prepare in advance for task switching. 


¢ The operating system can take the responsibility for saving the states as part of the task switch process, but 
delay the restoring of the states until an instruction operating on the states is actually executed by the new 
task. See Section 13.4.1, “Using the TS Flag to Control the Saving of the x87 FPU and SSE State,” for more 
information. This approach is called lazy restore. 


The use of lazy restore mechanism in context switches is not recommended when XSAVE feature set is used to 
save/restore states for the following reasons. 


— With XSAVE feature set, Intel processors have optimizations in place to avoid saving the state components 
that are in their initial configurations or when they have not been modified since they were restored last. 
These optimizations eliminate the need for lazy restore. See section 13.5.4 in Intel® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 1. 


— Intel processors have power optimizations when state components are in their initial configurations. Use of 
lazy restore retains the non-initial configuration of the last thread and is not power efficient. 


— Not all extended states support lazy restore mechanisms. As such, when one or more such states are 
enabled it becomes very inefficient to use lazy restore as it results in two separate state restore, one in 
context switch for the states that does not support lazy restore and one in the #NM handler for states that 
support lazy restore. 


13.4.1 | Using the TS Flag to Control the Saving of the x87 FPU and SSE State 


The TS flag in control register CRO is provided to allow the operating system to delay saving/restoring the x87 FPU 
and SSE state until an instruction that actually accesses this state is encountered in a new task. When the TS flag 
is set, the processor monitors the instruction stream for x87 FPU, MMX, SSE instructions. When the processor 
detects one of these instructions, it raises a device-not-available exception (#NM) prior to executing the instruc- 
tion. The #NM exception handler can then be used to save the x87 FPU and SSE state for the previous task (using 
an FXSAVE, XSAVE, or XSAVEOPT instruction) and load the x87 FPU and SSE state for the current task (using an 
FXRSTOR or XRSOTR instruction). If the task never encounters an x87 FPU, MMX, or SSE instruction, the device- 
not-available exception will not be raised and a task state will not be saved/restored unnecessarily. 


NOTE 
The CRC32 and POPCNT instructions do not operate on the x87 FPU or SSE state. They operate on 
the general-purpose registers and are not involved with the techniques described above. 


The TS flag can be set either explicitly (by executing a MOV instruction to control register CRO) or implicitly (using 
the IA-32 architecture’s native task switching mechanism). When the native task switching mechanism is used, the 
processor automatically sets the TS flag on a task switch. After the device-not-available handler has saved the x87 
FPU and SSE state, it should execute the CLTS instruction to clear the TS flag. 


13.5 THE XSAVE FEATURE SET AND PROCESSOR EXTENDED STATE 
MANAGEMENT 


The architecture of XSAVE feature set is described in CHAPTER 13 of Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1. The XSAVE feature set includes the following: 


¢ An extensible data layout for existing and future processor state extensions. The layout of the XSAVE area 
extends from the 512-byte FXSAVE/FXRSTOR layout to provide compatibility and migration path from 
managing the legacy FKSAVE/FXRSTOR area. The XSAVE area is described in more detail in Section 13.4 of the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


¢ CPUID enhancements for feature enumeration. See Section 13.2 of the Intel® 64 and IA-32 Architectures 
Software Developer’s Manual, Volume 1. 
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* Control register enhancement and dedicated register for enabling each processor extended state. See Section 
13.3 of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


¢ Instructions to save state to and restore state from the XSAVE area. See Section 13.7 through Section 13.9 of 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


Operating systems can utilize XSAVE feature set to manage both FP/SSE state and processor extended states. 
CPUID leaf ODH enumerates XSAVE feature set related information. The following guidelines provide the steps an 
operating system needs to take to support legacy FP/SSE states and processor extended states. 


1. Check that the processor supports the XSAVE feature set 


2. Determine the set of XSAVE managed features that the operating system intends to enable and calculate the 
size of the buffer needed to save/restore the states during context switch and other flows 


Enable use of XSAVE feature set and XSAVE managed features 
Provide an initialization for the XSAVE managed feature state components 


Provide (if necessary) required exception handlers for exceptions generated each of the XSAVE managed 
features. 


13.5.1 Checking the Support for XSAVE Feature Set 


Support for XSAVE Feature set is enumerated in CPUID.1.ECX.XSAVE[bit 26]. Enumeration of this bit indicates that 
the processor supports XSAVE/XRSTOR instructions to manage state and XSETBV/XGETBV on XCRO to enable and 
get enabled states. An operating system needs to enable XSAVE feature set as described later. 


Additionally CPUID.(EAX=0DH, ECX=1).EAX enumerates additional XSAVE sub features such as optimized save, 
compaction and supervisor state support. The following table summarizes XSAVE sub features. Once an operating 
system enables XSAVE feature set, all the sub-features enumerated are also available. There is no need to enable 
each additional sub feature. 


Table 13-3. CPUID.(EAX=ODH, ECX=1) EAX Bit Assignment 


EAX Bit Position Meaning 
0 If set, indicates availability of the XSAVEOPT instruction. 
1 If set, indicates availability of the XSAVEC instruction and the corresponding compaction enhancements 
to the legacy XRSTOR instruction. 
2 If set, indicates support for execution of XGETBV with ECX=1. This execution returns the state-compo- 


nent bitmap XINUSE. If XINUSE[i] = 0, state component i is in its initial configuration. Execution of 
XSETBV with ECX=1 causes a #GP. 


3 If set, indicates support for XSAVES/XRSTORS and IA32_XSS MSR 
31:4 Reserved 


13.5.2 Determining the XSAVE Managed Feature States And The Required Buffer Size 


Each XSAVE managed feature has one or more state components associated with it. An operating system policy 
needs to determine the XSAVE managed features to support and determine the corresponding state components to 
enable. When determining the XSAVE managed features to support, operating system needs to take into account 
the dependencies between them (e.g. AVX feature depends on SSE feature). Similarly, when a XSAVE managed 
feature has more than one state component, all of them need to be enabled. Each logical processor enumerates 
supported XSAVE state components in CPUID.(EAX=0DH, ECX=0).EDX:EAX. An operating system may enable all 
or a subset of the state components enumerated by the processor based on the OS policy. 


The size of the memory buffer needed to save enabled XSAVE state components depends on whether the OS opts- 
in to use compacted format or not. Section 13.4.3 of the Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 1 describes the layout of the extended region of the XSAVE area. 
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13.5.3. Enable the Use Of XSAVE Feature Set And XSAVE State Components 


Operating systems need to enable the use of XSAVE feature set by writing to CR4.0SXSAVE[bit 18] to enable 
XSETBV/XGETBV instructions to access XCRO and to support processor extended state management using 
XSAVE/XRSTOR. When XSAVE feature set is enabled, all enumerated XSAVE sub features such as optimized save, 
compaction and supervisor state support are also enabled. Operating systems also need to enable the XSAVE state 
components in XCRO using XSETBV instruction. 


XSAVE state components can subsequently be disabled in XCRO. However, disabling state components of AVX or 
AVX-512 that are not in initial configuration may incur power and performance penalty on SSE and AVX instructions 
respectively. If AVX state is disabled when it is not in its initial configuration, subsequent SSE instructions may 
incur a penalty. If AVX-512 state is disabled when it is not in its initial configuration, subsequent SSE and AVX 
instructions may incur a penalty. It is recommended that the operating systems and VMM set AVX or AVX-512 state 
components to their initial configuration before disabling them. This can be achieved by one of the two methods 
below. 


¢ Using XRSTOR: Operating system or VMM can set the state of AVX or AVX-512 state components using XRSTOR 
instruction before disabling them in XCRO. 


¢ Using VZEROUPPER: Operating system or VMM can set AVX and AVX-512 state components to their initial 
configuration using VZEROUPPER instruction before disabling them in XCRO. Note that this will set both AVX 
and AVX-512 state components to their initial configuration. If the intent is to only disable AVX-512 state, 
Operating system or VMM will need to save AVX state before executing VZEROUPPER and restore it afterwards. 


13.5.4 Provide an Initialization for the XSAVE State Components 


The XSAVE header of a newly allocated XSAVE area should be initialized to all zeroes before saving context. An 
operating system may choose to establish beginning state-component values for a task by executing XRSTOR from 
an XSAVE area that the OS has configured. If it is desired to begin state component i in its initial configuration, the 
OS should clear bit i in the XSTATE_BV field in the XSAVE header; otherwise, it should set that bit and place the 
desired beginning value in the appropriate location in the XSAVE area. 


When a buffer is allocated for compacted size, software must ensure that the XCOMP_BV field is setup correctly 
before restoring from the buffer. Bit 63 of the XCOMP_BV field indicates that the save area is in the compacted 
format and the remaining bits indicate the states that have space allocated in the save area. If the buffer is first 
used to save the state in compacted format, then the save instructions will setup the XCOMP_BV field appropri- 
ately. If the buffer is first used to restore the state, then software must set up the XCOMP_BV field. 


13.5.5 Providing the Required Exception Handlers 


Instructions part of each XSAVE managed features may generate exceptions and operating system may need to 
enable such exceptions and provide handlers for them. Section 13.8 describes feature specific OS requirements for 
each XSAVE managed features. 


13.6 INTEROPERABILITY OF THE XSAVE FEATURE SET AND FXSAVE/FXRSTOR 


The FXSAVE instruction writes x87 FPU and SSE state information to a 512-byte FXSAVE save area. FXRSTOR 
restores the processor’s x87 FPU and SSE states from an FXSAVE area. The XSAVE features set supports x87 FPU 
and SSE states using the same layout as the FXSAVE area to provide interoperability of FXSAVE versus XSAVE, and 
FXRSTOR versus XRSTOR. The XSAVE feature set allows system software to manage SSE state independent of x87 
FPU states. Thus system software that had been using FXSAVE and FXRSTOR to manage x87 FPU and SSE states 
can transition to using the XSAVE feature set to manage x87 FPU, SSE and other processor extended states in a 
systematic and forward-looking manner. See Section 10.5 and Chapter 13 of the Inte/® 64 and IA-32 Architectures 
Software Developer’s Manual, Volume 1 for more details. 


System software can implement forward-looking processor extended state management using the XSAVE feature 
set. In this case, system software must specify the bit vector mask in EDX:EAX appropriately when executing 
XSAVE/XRSTOR instructions. 
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For instance, the OS can supply instructions in the XSAVE feature set with a bit vector in EDX:EAX with the two 
least significant bits (corresponding to x87 FPU and SSE state) equal to 0. Then, the XSAVE instruction will not 
write the processor’s x87 FPU and SSE state into memory. Similarly, the XRSTOR instruction executed with a value 
in EDX:EAX with the least two significant bit equal to 0 will not restore nor initialize the processor’s x87 FPU and 
SSE state. 


The processor’s action as a result of executing XRSTOR is given in Section 13.8 of the Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 1. The instruction may be used to initialize x87 FPU or XMM regis- 
ters. When the MXCSR register is updated from memory, reserved bit checking is enforced. The saving/restoring of 
MXCSR is bound to the SSE state, independent of the x87 FPU state. The action of XSAVE is given in Section 13.7 
of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


13.7. THE XSAVE FEATURE SET AND PROCESSOR SUPERVISOR STATE 
MANAGEMENT 


Supervisor state is a processor state that is only accessible in ring 0. An extension to XSAVE feature set, enumer- 
ated by CPUID.(EAX=0DH, ECX=1).EAX[bit 3] allows the management of the supervisor states using XSAVE 

feature set. See Chapter 13 of Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 for the 
details of the supervisor state XSAVE feature set extension. The supervisor state extension includes the following: 


¢ CPUID enhancements to enumerate the set of supervisor states and their sizes that can be managed by XSAVE 
feature set. 


¢ Anew MSR IA32_XSS to enable XSAVE feature set to manage one or more enumerated supervisor states. 


¢ Anew pair of privileged save/restore instructions, XSAVES and XRSTORS, to save/restore supervisor states 
along with other XSAVE managed feature states. 


The guidelines to enable XSAVE feature set to manage supervisor state are very similar to the steps outlines in 
Section 13.6 with the differences outline below. The set of supervisor states that can be managed by XSAVE feature 
set is enumerated in (EAX=ODH, ECX=1).EDX:ECX. XSAVE managed supervisor states are enabled in IA32_XSS 
MSR instead of XCRO control register. There are semantic differences between user states enabled in XCRO and 
supervisor state enabled in IA32_XSS MSR. A supervisor state enabled in IA32_XSS MSR: 


¢ May be accessed via other mechanisms such as RDMSR/WRMSR even when they are not enabled in IA32_XSS 
MSR. Enabling a supervisor state in the IA32_XSS MSR merely indicates that the state can be saved/restored 
using XSAVES/XRSTORS instructions. 


° May have side effects when saving/restoring the state such as disabling/enabling feature associated with the 
state. This behavior is feature specific and will be documented along with the feature description. 


¢ May generate faults when saving/restoring the state. XSAVES/XRSTORS will follow the faulting behavior of 
RDMSR/WRMSR respectively if the corresponding state is also accessible using RDMSR/WRMSR. 


* XRSTORS may fault when restoring the state for supervisor features that are already enabled via feature 
specific mechanisms. This behavior is feature specific and will be documented along with the feature 
description. 


When a supervisor state is disabled via a feature specific mechanism, the state does not automatically get marked 
as INIT. Hence XSAVES/XRSTORS will continue to save/restore the state subject to available optimizations. If the 
software does not intend to preserve the state when it disables the feature, it should initialize it to hardware INIT 
value with XRSTORS instruction so that XSAVES/XRSTORS perform optimally for that state. 


13.8 SYSTEM PROGRAMMING FOR XSAVE MANAGED FEATURES 


This section describes system programming requirement for each XSAVE managed features that are feature 
specific such as exception handling. 
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13.8.1. Intel® Advanced Vector Extensions (Intel® AVX) 


Intel AVX instructions comprises of 256-bit and 128-bit instructions that operates on 256-bit YMM registers. The 
XSAVE feature set allows software to save and restore the state of these registers. See Chapter 13 of the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 1. 


For processors that support YMM states, the YMM state exists in all operating modes. However, the available 
instruction interfaces to access YMM states may vary in different modes. 


Operating systems must use the XSAVE feature set for YMM state management. The XSAVE feature set also 
provides flexible and efficient interface to manage XMM/MXCSR states and x87 FPU states in conjunction with 
newer processor extended states like YMM states. Operating systems may need to be aware of the following when 
supporting AVX. 


¢ Saving/Restoring AVX state in non-compacted format without SSE state will also save/restore MXCSR even 
though MXCSR is not part of AVX state. This does not happen when compacted format is used. 


¢ Few AVX instructions such as VZEROUPPER/VZEROALL may operate on future expansion of YMM registers. 


An operating system must enable its YMM state management to support AVX and any 256-bit extensions that 
operate on YMM registers. Otherwise, an attempt to execute an instruction in AVX extensions (including an 
enhanced 128-bit SIMD instructions using VEX encoding) will cause a #UD exception. 


AVX instructions may generate SIMD floating-point exceptions. An OS must enable SIMD floating-point exception 
support by setting CR4.0OSXMMEXCPT[bit 10]=1. 


13.8.2 — Intel® Advanced Vector Extensions 512 (Intel® AVX-512) 


Intel AVX-512 instructions are encoded using EVEX prefix. The EVEX encoding scheme can support 512-bit, 256- 
bit and 128-bit instructions that operate on opmask, ZMM, YMM and XMM registers. 


For processors that support the Intel AVX-512 family of instructions, the extended processor states (ZMM and 
opmask registers) exist in all operating modes. However, the access to these states may vary in different modes. 
The processor's support for instruction extensions that employ EVEX prefix encoding is independent of the 
processor's support for using XSAVE feature set on those states. 


Instructions requiring EVEX prefix encoding are generally supported in 64-bit, 32-bit modes, and 16-bit protected 
mode. They are not supported in Real mode, Virtual-8086 mode or entering into SMM mode. Note that bits MAXVL- 
1:256 (511:256) of ZMM register state are maintained across transitions into and out of these modes. Because the 
XSAVE feature set instruction can operate in all operating modes, it is possible that the processor's ZMM register 
state can be modified by software in any operating mode by executing XRSTOR. 


Operating systems must use the XSAVE/XRSTOR/XSAVEOPT instructions for ZMM and opmask state management. 
An OS must enable its ZMM and opmask state management to support Intel AVX-512 Foundation instructions. 
Otherwise, an attempt to execute an instruction in Intel AVX-512 Foundation instructions (including a scalar 128- 
bit SIMD instructions using EVEX encoding) will cause a #UD exception. An operating system, which enables the 
AVX-512 state to support Intel AVX-512 Foundation instructions, is also sufficient to support the rest of the Intel 
AVX-512 family of instructions. Note that even though ZMM8-ZMM31 are not accessible in 32 bit mode, a 32 bit OS 
is still required to allocate memory for the entire ZMM state. 


Intel AVX-512 Foundation instructions may generate SIMD floating-point exceptions. An OS must enable SIMD 
floating point exception support by setting CR4.OSXMMEXCPT[bit 10]=1. 
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CHAPTER 14 
POWER AND THERMAL MANAGEMENT 


This chapter describes facilities of Intel 64 and IA-32 architecture used for power management and thermal moni- 
toring. 


14.1. ENHANCED INTEL SPEEDSTEP® TECHNOLOGY 


Enhanced Intel SpeedStep® Technology was introduced in the Pentium M processor. The technology enables the 
management of processor power consumption via performance state transitions. These states are defined as 
discrete operating points associated with different voltages and frequencies. 


Enhanced Intel SpeedStep Technology differs from previous generations of Intel SpeedStep® Technology in two 
ways: 


* Centralization of the control mechanism and software interface in the processor by using model-specific 
registers. 


¢ Reduced hardware overhead; this permits more frequent performance state transitions. 


Previous generations of the Intel SpeedStep Technology require processors to be a deep sleep state, holding off bus 
master transfers for the duration of a performance state transition. Performance state transitions under the 
Enhanced Intel SpeedStep Technology are discrete transitions to a new target frequency. 


Support is indicated by CPUID, using ECX feature bit 07. Enhanced Intel SpeedStep Technology is enabled by 
setting IA32_MISC_ENABLE MSR, bit 16. On reset, bit 16 of IA32_MISC_ENABLE MSR is cleared. 


14.1.1. Software Interface For Initiating Performance State Transitions 


State transitions are initiated by writing a 16-bit value to the IA32_PERF_CTL register, see Figure 14-2. If a transi- 
tion is already in progress, transition to a new value will subsequently take effect. 


Reads of IA32_PERF_CTL determine the last targeted operating point. The current operating point can be read from 
IA32_PERF_STATUS. IA32_PERF_STATUS is updated dynamically. 


The 16-bit encoding that defines valid operating points is model-specific. Applications and performance tools are 
not expected to use either IA32_PERF_CTL or IA32_PERF_STATUS and should treat both as reserved. Performance 
monitoring tools can access model-specific events and report the occurrences of state transitions. 


14.2 P-STATE HARDWARE COORDINATION 


The Advanced Configuration and Power Interface (ACPI) defines performance states (P-states) that are used to 
facilitate system software's ability to manage processor power consumption. Different P-states correspond to 
different performance levels that are applied while the processor is actively executing instructions. Enhanced Intel 
SpeedStep Technology supports P-states by providing software interfaces that control the operating frequency and 
voltage of a processor. 


With multiple processor cores residing in the same physical package, hardware dependencies may exist for a 
subset of logical processors on a platform. These dependencies may impose requirements that impact the coordi- 
nation of P-state transitions. As a result, multi-core processors may require an OS to provide additional software 
support for coordinating P-state transitions for those subsets of logical processors. 


ACPI firmware can choose to expose P-states as dependent and hardware-coordinated to OS power management 
(OSPM) policy. To support OSPMs, multi-core processors must have additional built-in support for P-state hardware 
coordination and feedback. 


Intel 64 and IA-32 processors with dependent P-states amongst a subset of logical processors permit hardware 
coordination of P-states and provide a hardware-coordination feedback mechanism using IA32_MPERF MSR and 
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IA32_APERF MSR. See Figure 14-1 for an overview of the two 64-bit MSRs and the bullets below for a detailed 
description. 


63 0 63 0 


IA32_MPERF (Addr: E7H) IA32_APERF (Addr: E8H) 


Figure 14-1. [A32_MPERF MSR and IA32_APERF MSR for P-state Coordination 


¢ Use CPUID to check the P-State hardware coordination feedback capability bit. CPUID.O6H.ECX[Bit 0] = 1 
indicates IA32_MPERF MSR and IA32_APERF MSR are present. 

¢ IA32_MPERF MSR (E7H) increments in proportion to a fixed frequency, which is configured when the processor 
is booted. 

¢ TA32_APERF MSR (E8H) increments in proportion to actual performance, while accounting for hardware coordi- 
nation of P-state and TM1/TM2; or software initiated throttling. 

¢ The MSRs are per logical processor; they measure performance only when the targeted processor is in the CO 
state. 


° Only the IA32_APERF/IA32_MPERF ratio is architecturally defined; software should not attach meaning to the 
content of the individual of IA32_APERF or IA32_MPERF MSRs. 


¢ When either MSR overflows, both MSRs are reset to zero and continue to increment. 


° Both MSRs are full 64-bits counters. Each MSR can be written to independently. However, software should 
follow the guidelines illustrated in Example 14-1. 


If P-states are exposed by the BIOS as hardware coordinated, software is expected to confirm processor support 

for P-state hardware coordination feedback and use the feedback mechanism to make P-state decisions. The OSPM 
is expected to either save away the current MSR values (for determination of the delta of the counter ratio at a later 
time) or reset both MSRs (execute WRMSR with 0 to these MSRs individually) at the start of the time window used 
for making the P-state decision. When not resetting the values, overflow of the MSRs can be detected by checking 
whether the new values read are less than the previously saved values. 


Example 14-1 demonstrates steps for using the hardware feedback mechanism provided by IA32_APERF MSR and 
IA32_MPERF MSR to determine a target P-state. 


Example 14-1. Determine Target P-state From Hardware Coordinated Feedback 


DWORD PercentBusy; // Percentage of processor time not idle. 
// Measure “PercentBusy” during previous sampling window. 
// Typically, “PercentBusy” is measure over a time scale suitable for 
// power management decisions 
// 
// RDMSR of MCNT and ACNT should be performed without delay. 
// Software needs to exercise care to avoid delays between 
// the two RDMSRs (for example, interrupts). 
MCNT = RDMSR(IA32_MPERF); 
ACNT = RDMSR(IA32_APERF); 


// PercentPerformance indicates the percentage of the processor 
// that is in use. The calculation is based on the PercentBusy, 

// that is the percentage of processor time not idle and the P-state 
// hardware coordinated feedback using the ACNT/MCNT ratio. 

// Note that both values need to be calculated over the same 
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// time window. 
PercentPerformance = PercentBusy * (ACNT/MCNT); 


// This example does not cover the additional logic or algorithms 
// necessary to coordinate multiple logical processors to a target P-state. 


TargetPstate = FindPstate(PercentPerformance); 


if (TargetPstate # currentPstate) { 
SetPState(TargetPstate); 
} 
// \WRMSR of MCNT and ACNT should be performed without delay. 
// Software needs to exercise care to avoid delays between 
// the two WRMSRs (for example, interrupts). 
WRMSR(IA32_MPERF, 0); 
WRMSR(IA32_APERF, 0); 


14.3. SYSTEM SOFTWARE CONSIDERATIONS AND OPPORTUNISTIC PROCESSOR 
PERFORMANCE OPERATION 


An Intel 64 processor may support a form of processor operation that takes advantage of design headroom to 
opportunistically increase performance. The Intel® Turbo Boost Technology can convert thermal headroom into 
higher performance across multi-threaded and single-threaded workloads. The Intel® Dynamic Acceleration Tech- 
nology feature can convert thermal headroom into higher performance if only one thread is active. 


14.3.1 Intel® Dynamic Acceleration Technology 


The Intel Core 2 Duo processor T 7700 introduces Intel Dynamic Acceleration Technology. Intel Dynamic Accelera- 
tion Technology takes advantage of thermal design headroom and opportunistically allows a single core to operate 
at a higher performance level when the operating system requests increased performance. 


14.3.2 System Software Interfaces for Opportunistic Processor Performance Operation 


Opportunistic processor performance operation, applicable to Intel Dynamic Acceleration Technology and Intel® 
Turbo Boost Technology, has the following characteristics: 


¢ Atransition from a normal state of operation (e.g. Intel Dynamic Acceleration Technology/Turbo mode 
disengaged) to a target state is not guaranteed, but may occur opportunistically after the corresponding enable 
mechanism is activated, the headroom is available and certain criteria are met. 


¢ The opportunistic processor performance operation is generally transparent to most application software. 


° System software (BIOS and Operating system) must be aware of hardware support for opportunistic processor 
performance operation and may need to temporarily disengage opportunistic processor performance operation 
when it requires more predictable processor operation. 


¢ When opportunistic processor performance operation is engaged, the OS should use hardware coordination 
feedback mechanisms to prevent un-intended policy effects if it is activated during inappropriate situations. 


14.3.2.1 Discover Hardware Support and Enabling of Opportunistic Processor Performance Operation 


If an Intel 64 processor has hardware support for opportunistic processor performance operation, the power-on 
default state of IA32_MISC_ENABLE[38] indicates the presence of such hardware support. For Intel 64 processors 
that support opportunistic processor performance operation, the default value is 1, indicating its presence. For 
processors that do not support opportunistic processor performance operation, the default value is 0. The power- 
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on default value of IA32_MISC_ENABLE[38] allows BIOS to detect the presence of hardware support of opportu- 
nistic processor performance operation. 


IA32_MISC_ENABLE[38] is shared across all logical processors in a physical package. It is written by BIOS during 
platform initiation to enable/disable opportunistic processor performance operation in conjunction of OS power 
management capabilities, see Section 14.3.2.2. BIOS can set IA32_MISC_ENABLE[38] with 1 to disable opportu- 
nistic processor performance operation; it must clear the default value of IA32_MISC_ENABLE[38] to 0 to enable 
opportunistic processor performance operation. OS and applications must use CPUID leaf O6H if it needs to detect 
processors that have opportunistic processor performance operation enabled. 


When CPUID is executed with EAX = 06H on input, Bit 1 of EAX in Leaf 06H (i.e. CPUID.06H:EAX[1]) indicates 
opportunistic processor performance operation, such as Intel Dynamic Acceleration Technology, has been enabled 
by BIOS. 


Opportunistic processor performance operation can be disabled by setting bit 38 of IA32_MISC_ENABLE. This 
mechanism is intended for BIOS only. If IA32_MISC_ENABLE[38] is set, CPUID.O6H:EAX[1] will return 0. 


14.3.2.2 OS Control of Opportunistic Processor Performance Operation 


There may be phases of software execution in which system software cannot tolerate the non-deterministic aspects 
of opportunistic processor performance operation. For example, when calibrating a real-time workload to make a 
CPU reservation request to the OS, it may be undesirable to allow the possibility of the processor delivering 
increased performance that cannot be sustained after the calibration phase. 


System software can temporarily disengage opportunistic processor performance operation by setting bit 32 of the 
IA32_PERF_CTL MSR (0199H), using a read-modify-write sequence on the MSR. The opportunistic processor 
performance operation can be re-engaged by clearing bit 32 in IA32_PERF_CTL MSR, using a read-modify-write 
sequence. The DISENAGE bit in IA32_PERF_CTL is not reflected in bit 32 of the IA32_PERF_STATUS MSR (0198H), 
and it is not shared between logical processors in a physical package. In order for OS to engage Intel Dynamic 
Acceleration Technology/Turbo mode, the BIOS must: 


¢ Enable opportunistic processor performance operation, as described in Section 14.3.2.1. 
¢ Expose the operating points associated with Intel Dynamic Acceleration Technology/Turbo mode to the OS. 
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Reserved Reserved 


Intel® Dynamic Acceleration Technology / Turbo DISENGAGE 


Enhanced Intel Speedstep® Technology Transition Target 


Figure 14-2. IA32_PERF_CTL Register 


14.3.2.3. Required Changes to OS Power Management P-State Policy 


Intel Dynamic Acceleration Technology and Intel Turbo Boost Technology can provide opportunistic performance 
greater than the performance level corresponding to the Processor Base frequency of the processor (see CPUID's 
processor frequency information). System software can use a pair of MSRs to observe performance feedback. Soft- 
ware must query for the presence of IA32_APERF and IA32_MPERF (see Section 14.2). The ratio between 
IA32_APERF and IA32_MPERF is architecturally defined and a value greater than unity indicates performance 
increase occurred during the observation period due to Intel Dynamic Acceleration Technology. Without incorpo- 
rating such performance feedback, the target P-state evaluation algorithm can result in a non-optimal P-state 
target. 
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There are other scenarios under which OS power management may want to disable Intel Dynamic Acceleration 
Technology, some of these are listed below: 


¢ When engaging ACPI defined passive thermal management, it may be more effective to disable Intel Dynamic 
Acceleration Technology for the duration of passive thermal management. 


¢ When the user has indicated a policy preference of power savings over performance, OS power management 
may want to disable Intel Dynamic Acceleration Technology while that policy is in effect. 


14.3.3. Intel® Turbo Boost Technology 


Intel Turbo Boost Technology is supported in Intel Core i7 processors and Intel Xeon processors based on Intel® 
microarchitecture code name Nehalem. It uses the same principle of leveraging thermal headroom to dynamically 
increase processor performance for single-threaded and multi-threaded/multi-tasking environment. The program- 
ming interface described in Section 14.3.2 also applies to Intel Turbo Boost Technology. 


14.3.4 Performance and Energy Bias Hint Support 


Intel 64 processors may support additional software hint to guide the hardware heuristic of power management 
features to favor increasing dynamic performance or conserve energy consumption. 


Software can detect the processor's capability to support the performance-energy bias preference hint by exam- 
ining bit 3 of ECX in CPUID leaf 6. The processor supports this capability if CPUID.O6H:ECX.SETBH[bit 3] is set and 
it also implies the presence of a new architectural MSR called IA32_ENERGY_PERF_BIAS (1BOH). 


Software can program the lowest four bits of IA32_ENERGY_PERF_BIAS MSR with a value from 0 - 15. The values 
represent a sliding scale, where a value of 0 (the default reset value) corresponds to a hint preference for highest 
performance and a value of 15 corresponds to the maximum energy savings. A value of 7 roughly translates into a 
hint to balance performance with energy consumption. 


63 43 0 


Reserved 


Energy Policy Preference Hint 


Figure 14-3. IA32_ENERGY_PERF_BIAS Register 


The layout of IA32_ENERGY_PERF_BIAS is shown in Figure 14-3. The scope of IA32_ENERGY_PERF_BIAS is per 
logical processor, which means that each of the logical processors in the package can be programmed with a 
different value. This may be especially important in virtualization scenarios, where the performance / energy 
requirements of one logical processor may differ from the other. Conflicting “hints” from various logical processors 
at higher hierarchy level will be resolved in favor of performance over energy savings. 


Software can use whatever criteria it sees fit to program the MSR with an appropriate value. However, the value 
only serves as a hint to the hardware and the actual impact on performance and energy savings is model specific. 


14.4 HARDWARE-CONTROLLED PERFORMANCE STATES (HWP) 


Intel processors may contain support for Hardware-Controlled Performance States (HWP), which autonomously 
selects performance states while utilizing OS supplied performance guidance hints. The Enhanced Intel Speed- 
Step® Technology provides a means for the OS to control and monitor discrete frequency-based operating points 
via the IA32_PERF_CTL and IA32_PERF_STATUS MSRs. 
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In contrast, HWP is an implementation of the ACPI-defined Collaborative Processor Performance Control (CPPC), 
which specifies that the platform enumerates a continuous, abstract unit-less, performance value scale that is not 
tied to a specific performance state / frequency by definition. While the enumerated scale is roughly linear in terms 
of a delivered integer workload performance result, the OS is required to characterize the performance value range 
to comprehend the delivered performance for an applied workload. 


When HWP is enabled, the processor autonomously selects performance states as deemed appropriate for the 
applied workload and with consideration of constraining hints that are programmed by the OS. These OS-provided 
hints include minimum and maximum performance limits, preference towards energy efficiency or performance, 
and the specification of a relevant workload history observation time window. The means for the OS to override 
HWP's autonomous selection of performance state with a specific desired performance target is also provided, 
however, the effective frequency delivered is subject to the result of energy efficiency and performance optimiza- 
tions. 


14.4.1 HWP Programming Interfaces 


The programming interfaces provided by HWP include the following: 
¢ The CPUID instruction allows software to discover the presence of HWP support in an Intel processor. Specifi- 


cally, execute CPUID instruction with EAX=O6H as input will return 5 bit flags covering the following aspects in 
bits 7 through 11 of CPUID.O6H:EAX: 


— Availability of HWP baseline resource and capability, CPUID.06H:EAX[bit 7]: If this bit is set, HWP provides 
several new architectural MSRs: IA32_PM_ENABLE, IA32_HWP_CAPABILITIES, IA32_HWP_REQUEST, 
IA32_HWP_STATUS. 


— Availability of HWP Notification upon dynamic Guaranteed Performance change, CPUID.06H:EAX[bit 8]: If 
this bit is set, HWP provides IA32_HWP_INTERRUPT MSR to enable interrupt generation due to dynamic 
Performance changes and excursions. 


— Availability of HWP Activity window control, CPUID.O6H:EAX[bit 9]: If this bit is set, HWP allows software to 
program activity window in the IA32_ HWP_REQUEST MSR. 


— Availability of HWP energy/performance preference control, CPUID.06H:EAX[bit 10]: If this bit is set, HWP 
allows software to set an energy/performance preference hint in the IA32_HWP_REQUEST MSR. 


— Availability of HWP package level control, CPUID.O6H:EAX[bit 11]:If this bit is set, HWP provides the 
IA32_HWP_REQUEST_PKG MSR to convey OS Power Management’s control hints for all logical processors 
in the physical package. 


Table 14-1. Architectural and Non-Architectural MSRs Related to HWP 


Address | Architectural Register Name Description 
770H Y 1A32_PM_ENABLE Enable/Disable HWP. 
771H Y IA32_HWP_CAPABILITIES Enumerates the HWP performance range (static and dynamic). 
772H Y IA32_HWP_REQUEST_PKG Conveys OSPM's control hints (Min, Max, Activity Window, Energy 


Performance Preference, Desired) for all logical processor in the 
physical package. 


773H Y IA32_HWP_INTERRUPT Controls HWP native interrupt generation (Guaranteed Performance 
changes, excursions). 

774H Y IA32_HWP_REQUEST Conveys OSPM's control hints (Min, Max, Activity Window, Energy 
Performance Preference, Desired) for a single logical processor. 

775H Y IA32_HWP_PECI_LREQUEST_INFO | Conveys embedded system controller requests to override some of 
the OS HWP Request settings via the PECI mechanism. 

777H Y IA32_HWP_STATUS Status bits indicating changes to Guaranteed Performance and 
excursions to Minimum Performance. 

19CH Y IA32_THERM_STATUS[bits 15:12] | Conveys reasons for performance excursions. 

64EH N MSR_PPERF Productive Performance Count. 
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¢ ~=Additionally, HWP may provide a non-architectural MSR, MSR_PPERF, which provides a quantitative metric to 
software of hardware’s view of workload scalability. This hardware’s view of workload scalability is implemen- 
tation specific. 


14.4.2 Enabling HWP 
The layout of the IA32_PM_ENABLE MSR is shown in Figure 14-4. The bit fields are described below: 


63 10 


Reserved 


HWP_ENABLE 


Figure 14-4. IA32_PM_ENABLE MSR 


¢ HWP_ENABLE (bit 0, R/W10Once) — Software sets this bit to enable HWP with autonomous selection of 
processor P-States. When set, the processor will disregard input from the legacy performance control interface 
(IA32_PERF_CTL). Note this bit can only be enabled once from the default value. Once set, writes to the 
HWP_ENABLE bit are ignored. Only RESET will clear this bit. Default = zero (0). 

¢ Bits 63:1 are reserved and must be zero. 


After software queries CPUID and verifies the processor’s support of HWP, system software can write 1 to 
IA32_PM_ENABLE.HWP_ENABLE (bit 0) to enable hardware controlled performance states. The default value of 
IA32_PM_ENABLE MSR at power-on is 0, i.e. HWP is disabled. 


Additional MSRs associated with HWP may only be accessed after HWP is enabled, with the exception of 
IA32_HWP_INTERRUPT and MSR_PPERF. Accessing the IA32_ HWP_INTERRUPT MSR requires only HWP is present 
as enumerated by CPUID but does not require enabling HWP. 


IA32_PM_ENABLE is a package level MSR, i.e., writing to it from any logical processor within a package affects all 
logical processors within that package. 


14.4.3. HWP Performance Range and Dynamic Capabilities 


The OS reads the IA32_HWP_CAPABILITIES MSR to comprehend the limits of the HWP-managed performance 
range as well as the dynamic capability, which may change during processor operation. The enumerated perfor- 
mance range values reported by IA32_ HWP_CAPABILITIES directly map to initial frequency targets (prior to work- 
load-specific frequency optimizations of HWP). However the mapping is processor family specific. 


The layout of the IA32_HWP_CAPABILITIES MSR is shown in Figure 14-5. The bit fields are described below: 
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63 32 31 24 23 1615 87 0 


Reserved 


Lowest_Performance 


Most_Efficient_Performance 


Guaranteed_Performance 


Highest_Performance 


Figure 14-5. [A32_HWP_CAPABILITIES Register 


* Highest_Performance (bits 7:0, RO) — Value for the maximum non-guaranteed performance level. 


* Guaranteed_Performance (bits 15:8, RO) — Current value for the guaranteed performance level. This 
value can change dynamically as a result of internal or external constraints, e.g. thermal or power limits. 


° Most_Efficient_Performance (bits 23:16, RO) — Current value of the most efficient performance level. 
This value can change dynamically as a result of workload characteristics. 


° Lowest_Performance (bits 31:24, RO) — Value for the lowest performance level that software can program 
to IA32_HWP_REQUEST. 


° Bits 63:32 are reserved and must be zero. 


The value returned in the Guaranteed_Performance field is hardware's best-effort approximation of the avail- 
able performance given current operating constraints. Changes to the Guaranteed_Performance value will 
primarily occur due to a shift in operational mode. This includes a power or other limit applied by an external agent, 
e.g. RAPL (see Figure 14.10.1), or the setting of a Configurable TDP level (see model-specific controls related to 
Programmable TDP Limit in Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures 
Software Developer’s Manual, Volume 4.). Notification of a change to the Guaranteed_Performance occurs via 
interrupt (if configured) and the IA32_HWP_Status MSR. Changes to Guaranteed_Performance are indicated when 
a macroscopically meaningful change in performance occurs i.e. sustained for greater than one second. Conse- 
quently, notification of a change in Guaranteed Performance will typically occur no more frequently than once per 
second. Rapid changes in platform configuration, e.g. docking / undocking, with corresponding changes to a 
Configurable TDP level could potentially cause more frequent notifications. 


The value returned by the Most_Efficient_Performance field provides the OS with an indication of the practical 
lower limit for the IA32_HWP_REQUEST. The processor may not honor IA32_HWP_REQUEST.Maximum Perfor- 
mance settings below this value. 


14.4.4 Managing HWP 


14.4.4.1 IA32_HWP_REQUEST MSR (Address: 774H Logical Processor Scope) 


Typically, the operating system controls HWP operation for each logical processor via the writing of control hints / 
constraints to the IA32_HWP_REQUEST MSR. The layout of the IA32_ HWP_REQUEST MSR is shown in Figure 14-6. 
The bit fields are described below Figure 14-6. 


Operating systems can control HWP by writing both IA32_HWP_REQUEST and IA32_HWP_REQUEST_PKG MSRs 
(see Section 14.4.4.2). Five valid bits within the IA32_HWP_REQUEST MSR let the operating system flexibly select 
which of its five hint / constraint fields should be derived by the processor from the IA32_ HWP_REQUEST MSR and 
which should be derived from the IA32_HWP_REQUEST_PKG MSR. These five valid bits are supported if 
CPUID[6].EAX[17] is set. 
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When the IA32_HWP_REQUEST MSR Package Control bit is set, any valid bit that is NOT set indicates to the 
processor to use the respective field value from the IA32_ HWP_REQUEST_PKG MSR. Otherwise, the values are 
derived from the IA32_ HWP_REQUEST MSR. The valid bits are ignored when the IA32_HWP_REQUEST MSR 
Package Control bit is zero. 


63 62 61 60 59 43 42 41 32 31 24 23 1615 87 0 
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Activity_Window Valid 
Package_Control 
Activity_Window 


Energy_Performance_Preference 
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Figure 14-6. IA32_HWP_REQUEST Register 


¢ Minimum_Performance (bits 7:0, RW) — Conveys a hint to the HWP hardware. The OS programs the 
minimum performance hint to achieve the required quality of service (QOS) or to meet a service level 
agreement (SLA) as needed. Note that an excursion below the level specified is possible due to hardware 
constraints. The default value of this field is IA32_HWP_CAPABILITIES.Lowest_Performance. 


¢ Maximum_Performance (bits 15:8, RW) — Conveys a hint to the HWP hardware. The OS programs this 
field to limit the maximum performance that is expected to be supplied by the HWP hardware. Excursions 
above the limit requested by OS are possible due to hardware coordination between the processor cores and 
other components in the package. The default value of this field is 
IA32_HWP_CAPABILITIES.Highest_Performance. 


° Desired_Performance (bits 23:16, RW) — Conveys a hint to the HWP hardware. When set to zero, 
hardware autonomous selection determines the performance target. When set to a non-zero value (between 
the range of Lowest_Performance and Highest_Performance of IA32_HWP_CAPABILITIES) conveys an explicit 
performance request hint to the hardware; effectively disabling HW Autonomous selection. The 
Desired_Performance input is non-constraining in terms of Performance and Energy Efficiency optimizations, 
which are independently controlled. The default value of this field is 0. 


¢ Energy_Performance_Preference (bits 31:24, RW) — Conveys a hint to the HWP hardware. The OS may 
write a range of values from 0 (performance preference) to OFFH (energy efficiency preference) to influence 
the rate of performance increase /decrease and the result of the hardware's energy efficiency and performance 
optimizations. The default value of this field is 80H. Note: If CPUID.O6H:EAX[bit 10] indicates that this field is 
not supported, HWP uses the value of the IA32_ENERGY_PERF_BIAS MSR to determine the energy efficiency / 
performance preference. 


° Activity_Window (bits 41:32, RW) — Conveys a hint to the HWP hardware specifying a moving workload 
history observation window for performance/frequency optimizations. If 0, the hardware will determine the 
appropriate window size. When writing a non-zero value to this field, this field is encoded in the format of bits 
38:32 as a 7-bit mantissa and bits 41:39 as a 3-bit exponent value in powers of 10. The resultant value is in 
microseconds. Thus, the minimal/maximum activity window size is 1 microsecond/1270 seconds. Combined 
with the Energy_Performance_Preference input, Activity_Window influences the rate of performance increase 
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/ decrease. This non-zero hint only has meaning when Desired_Performance = O. The default value of this field 
is 0. 

¢ Package_Control (bit 42, RW) — When set, causes this logical processor's IA32_HWP_REQUEST control 
inputs to be derived from the IA32_HWP_REQUEST_PKG MSR. 


° Bits 58:43 are reserved and must be zero. 


° Activity_Window Valid (bit 59, RW) — When set, indicates to the processor to derive the Activity Window 
field value from the IA32_ HWP_REQUEST MSR even if the package control bit is set. Otherwise, derive it from 
the IA32_HWP_REQUEST_PKG MSR. The default value of this field is 0. 


¢ EPP Valid (bit 60, RW) — When set, indicates to the processor to derive the EPP field value from the 
IA32_HWP_REQUEST MSR even if the package control bit is set. Otherwise, derive it from the 
IA32_HWP_REQUEST_PKG MSR. The default value of this field is 0. 


° Desired Valid (bit 61, RW) — When set, indicates to the processor to derive the Desired Performance field 
value from the IA32_ HWP_REQUEST MSR even if the package control bit is set. Otherwise, derive it from the 
IA32_HWP_REQUEST_PKG MSR. The default value of this field is 0. 


¢ Maximum Valid (bit 62, RW) — When set, indicates to the processor to derive the Maximum Performance 
field value from the IA32_ HWP_REQUEST MSR even if the package control bit is set. Otherwise, derive it from 
the IA32_HWP_REQUEST_PKG MSR. The default value of this field is 0. 


° Minimum Valid (bit 63, RW) — When set, indicates to the processor to derive the Minimum Performance field 
value from the IA32_HWP_REQUEST MSR even if the package control bit is set. Otherwise, derive it from the 
IA32_HWP_REQUEST_PKG MSR. The default value of this field is 0. 


The HWP hardware clips and resolves the field values as necessary to the valid range. Reads return the last value 
written not the clipped values. 


Processors may support a subset of IA32_ HWP_REQUEST fields as indicated by CPUID. Reads of non-supported 
fields will return 0. Writes to non-supported fields are ignored. 


The OS may override HWP's autonomous selection of performance state with a specific performance target by 
setting the Desired_Performance field to a non-zero value, however, the effective frequency delivered is subject to 
the result of energy efficiency and performance optimizations, which are influenced by the Energy Performance 
Preference field. 


Software may disable all hardware optimizations by setting Minimum_Performance = Maximum_Performance 
(subject to package coordination). 


Note: The processor may run below the Minimum_Performance level due to hardware constraints including: power, 
thermal, and package coordination constraints. The processor may also run below the Minimum_Performance level 
for short durations (few milliseconds) following C-state exit, and when Hardware Duty Cycling (see Section 14.5) is 
enabled. 


When the JIA32_HWP_REQUEST MSR is set to fast access mode, writes of this MSR are posted, i.e., the WRMSR 
instruction retires before the data reaches its destination within the processor. It may retire even before all 
preceding IA stores are globally visible, i.e., it is not an architecturally serializing instruction anymore (no store 
fence). A new CPUID bit indicates this new characteristic of the IA32_HWP_REQUEST MSR (see Section 14.4.8 for 
additional details). 
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14.4.4.2 1A32_HWP_REQUEST_PKG MSR (Address: 772H Package Scope) 
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Maximum_Performance 
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Figure 14-7. IA32_HWP_REQUEST_PKG Register 


The structure of the IA32_HWP_REQUEST_PKG MSR (package-level) is identical to the IA32_HWP_REQUEST MSR 
with the exception of the the Package Control bit field and the five valid bit fields, which do not exist in the 
IA32_HWP_REQUEST_PKG MSR. Field values written to this MSR apply to all logical processors within the physical 
package with the exception of logical processors whose IA32_HWP_REQUEST.Package Control field is clear (zero). 
Single P-state Control mode is only supported when IA32_HWP_REQUEST_PKG is not supported. 


14.4.4.3. 1IA32_HWP_PECI_REQUEST_INFO MSR (Address 775H Package Scope) 


When an embedded system controller is integrated in the platform, it can override some of the OS HWP Request 
settings via the PECI mechanism. PECI initiated settings take precedence over the relevant fields in the 
IA32_HWP_REQUEST MSR and in the IA32_HWP_REQUEST_PKG MSR, irrespective of the Package Control bit or 
the Valid Bit values described above. PECI can independently control each of: Minimum Performance, Maximum 
Performance and EPP fields. This MSR contains both the PECI induced values and the control bits that indicate 
whether the embedded controller actually set the processor to use the respective value. 


PECI override is supported if CPUID[6].EAX[16] is set. 
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Figure 14-8. IA32_HWP_PECI_REQUEST_INFO MSR 
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The layout of the IA32_HWP_PECI_REQUEST_INFO MSR is shown in Figure 14-8. This MSR is writable by the 
embedded controller but is read-only by software executing on the CPU. This MSR has Package scope. The bit fields 
are described below: 


¢ Minimum_Performance (bits 7:0, RO) — Used by the OS to read the latest value of PECI minimum 
performance input. 


¢ Maximum_Performance (bits 15:8, RO) — Used by the OS to read the latest value of PECI maximum 
performance input. 


¢ Bits 23:16 are reserved and must be zero. 


¢ Energy_Performance_Preference (bits 31:24, RO) — Used by the OS to read the latest value of PECI 
energy performance preference input. 


¢ Bits 59:32 are reserved and must be zero. 


¢ EPP_PECI_Override (bit 60, RO) — Indicates whether PECI if currently overriding the Energy Performance 
Preference input. If set(1), PECI is overriding the Energy Performance Preference input. If clear(0), OS has 
control over Energy Performance Preference input. 


¢ Bit 61 is reserved and must be zero. 


¢ Max_PECI_Override (bit 62, RO) — Indicates whether PECI if currently overriding the Maximum 
Performance input. If set(1), PECI is overriding the Maximum Performance input. If clear(0), OS has control 
over Maximum Performance input. 


¢ Min_PECI_Override (bit 63, RO) — Indicates whether PECI if currently overriding the Minimum Performance 
input. If set(1), PECI is overriding the Minimum Performance input. If clear(0), OS has control over Minimum 
Performance input. 


HWP Request Field Hierarchical Resolution 


HWP Request field resolution is fed by three MSRs: IA32_ HWP_REQUEST, IA32_HWP_REQUEST_PKG and 
IA32_HWP_PECI_REQUEST_INFO. The flow that the processor goes through to resolve which field value is chosen 
is shown below. 
For each of the two HWP Request fields; Desired and Activity Window: 
If IA32_HWP_REQUEST.PACKAGE_CONTROL = 1 and IA32_HWP_REQUEST. <field> valid bit = 0 
Resolved Field Value = IA32_HWP_REQUEST_PKG.<field> 
Else 
Resolved Field Value = IA32_HWP_REQUEST. <field> 
For each of the three HWP Request fields; Min, Max and EPP: 
If IA32_HWP_PECI_REQUEST_INFO.<field> PECI Override bit = 1 
Resolved Field Value = IA32_HWP_PECI_REQUEST_INFO. <field> 
Else if IA32_HWP_REQUEST.PACKAGE_CONTROL = 1 and IA32_ HWP_REQUEST. <field> valid bit = 0 
Resolved Field Value = IA32_HWP_REQUEST_PKG.<field> 
Else 
Resolved Field Value = IA32_HWP_REQUEST. <field> 


14.4.4.4 IA32_HWP_CTL MSR (Address: 776H Logical Processor Scope) 


IA32_HWP_CTL[0] controls the behavior of IA32_HWP_REQUEST Package Control [bit 42]. This control bit allows 
the IA32_ HWP_REQUEST MSR to stay in INIT mode most of the time (Control Bit is equal to its RESET value of 0) 
thus avoiding actual saving/restoring of the MSR contents when the OS adds it to the register set saved and 
restored by XSAVES/XRSTORS. 


* When IA32_HWP_CTL[0] = 0: 


— If IA32_HWP_REQUEST[42] = 0, the processor ignores all fields of the IA32_ HWP_REQUEST_PKG MSR and 
selects all HWP values (Min, Max, EPP, Desired, Activity Window) from the IA32_ HWP_REQUEST MSR. 


— If IA32_HWP_REQUEST[42] = 1, the processor selects the HWP values (Min, Max, EPP, Desired, Activity 
Window) either from the IA32_ HWP_REQUEST MSR or from the IA32_ HWP_REQUEST_PKG MSR according 
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to the values contained in the IA32_HWP_REQUEST MSR bit fields [bits 63:59]. See Section 14.4.4.1 for 
additional details. 


¢ When IA32_HWP_CTL[0] = 1, the behavior is reversed: 


— If IA32_HWP_REQUEST[42] = 1, the processor ignores all fields of the IA32_HWP_REQUEST_PKG MSR and 
selects all HWP values (Min, Max, EPP, Desired, Activity Window) from the IA32_HWP_REQUEST MSR. 


— If IA32_HWP_REQUEST[42] = 0, the processor selects the HWP values (Min, Max, EPP, Desired, Activity 
Window) either from the IA32_HWP_REQUEST MSR or from the IA32_HWP_REQUEST_PKG MSR according 
to the values contained in the IA32_ HWP_REQUEST MSR bit fields [bits 63:59]. See Section 14.4.4.1 for 
additional details. 


Section 14-2 summarizes the IA32_HWP_CTL MSR bit 0 control behavior. 


Table 14-2. I[A32_HWP_CTL MSR Bit 0 Behavior 
Field Description 


Threadrequest | Defines which HWP Request MSR is used, whether thread level or package level. When the package MSR is used, the 
PKG CTL thread MSR valid bits define which thread MSR fields override the package (default 0). 


meaning | 1q32_HWP_CTL[PKG_CTL_PLR] | IA32_HWP_REQUEST[PKG_CTL] HWP Request MSR Used 
0 0 |A32_HWP_REQUEST MSR 
0 1 IA32_HwP_REQUEST_PKG MSR 
1 0 IA32_HwP_REQUEST_PKG MSR 
1 1 |A32_HWP_REQUEST MSR 


This MSR is supported if CPUID[6].EAX[22] is set. 
If the IA32_PM_ENABLE[HWP_ENABLE] (bit 0 ) is not set, access to this MSR will generate a #GP fault. 


14.4.5 HWP Feedback 
The processor provides several types of feedback to the OS during HWP operation. 


The IA32_MPERF MSR and IA32_APERF MSR mechanism (see Section 14.2) allows the OS to calculate the resultant 
effective frequency delivered over a time period. Energy efficiency and performance optimizations directly impact 
the resultant effective frequency delivered. 


The layout of the IA32_ HWP_STATUS MSR is shown in Figure 14-9. It provides feedback regarding changes to 
IA32_HWP_CAPABILITIES.Guaranteed_Performance, IA32_HWP_CAPABILITIES.Highest_Performance, excur- 
sions to IA32_HWP_CAPABILITIES.Minimum_Performance, and PECI_Override entry/exit events. The bit fields are 
described below: 


¢ Guaranteed_Performance_Change (bit 0, RWCO) — If set (1), a change to Guaranteed_Performance has 
occurred. Software should query IA32_HWP_CAPABILITIES.Guaranteed_Performance value to ascertain the 
new Guaranteed Performance value and to assess whether to re-adjust HWP hints via IA32_ HWP_REQUEST. 
Software must clear this bit by writing a zero (0). 


e Bit 1 is reserved and must be zero. 


¢ Excursion_To_Minimum (bit 2, RWCO) — If set (1), an excursion to Minimum_Performance of 
IA32_HWP_REQUEST has occurred. Software must clear this bit by writing a zero (0). 


° Highest_Change (bit 3, RWCO) — If set (1), a change to Highest Performance has occurred. Software 
should query IA32_HWP_CAPABILITIES to ascertain the new Highest Performance value. Software must clear 
this bit by writing a zero (0). Interrupts upon Highest Performance change are supported if CPUID[6].EAX[15] 
is set. 


¢ PECI_Override_Entry (bit 4, RWCO) — If set (1), an embedded/management controller has started a PECI 
override of one or more OS control hints (Min, Max, EPP) specified in IA32_HWP_REQUEST or 
IA32_HWP_REQUEST_PKG. Software may query IA32_HWP_PECI_REQUEST_INFO MSR to ascertain which 
fields are now overridden via the PECI mechanism and what their values are (see Section 14.4.4.3 for 
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additional details). Software must clear this bit by writing a zero (0). Interrupts upon PECI override entry are 
supported if CPUID[6].EAX[16] is set. 


¢ PECI_Override_Exit (bit 5, RWCO) — If set (1), an embedded/management controller has stopped 
overriding one or more OS control hints (Min, Max, EPP) specified in IA32_HWP_REQUEST or 
IA32_HWP_REQUEST_PKG. Software may query IA32_HWP_PECI_REQUEST_INFO MSR to ascertain which 
fields are still overridden via the PECI mechanism and which fields are now back under software control (see 
Section 14.4.4.3 for additional details). Software must clear this bit by writing a zero (0). Interrupts upon PECI 
override exit are supported if CPUID[6].EAX[16] is set. 


° Bits 63:6 are reserved and must be zero. 


63 6543210 


Reserved 


PECI_Override_Exit 


PECI_Override_Entry 


Highest_Change 


Excursion_To_Minimum 


Reserved 


Guaranteed_Performance_Change 


Figure 14-9. |IA32_HWP_STATUS MSR 


The status bits of IA32_HWP_STATUS must be cleared (0) by software so that a new status condition change will 
cause the hardware to set the bit again and issue the notification. Status bits are not set for “normal” excursions, 
e.g., running below Minimum Performance for short durations during C-state exit. Changes to 
Guaranteed_Performance, Highest_Performance, excursions to Minimum_Performance, or PECI_Override 
entry/exit will occur no more than once per second. 


The OS can determine the specific reasons for a Guaranteed_Performance change or an excursion to 
Minimum_Performance in IA32_HWP_REQUEST by examining the associated status and log bits reported in the 
IA32_THERM_STATUS MSR. The layout of the IA32_ HWP_STATUS MSR that HWP uses to support software query of 
HWP feedback is shown in Figure 14-10. The bit fields of IA32_THERM_STATUS associated with HWP feedback are 
described below (Bit fields of IA32_THERM_STATUS unrelated to HWP can be found in Section 14.8.5.2). 
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63 32 31 27 «2322 «161514131211 109 8 7 6 5 362 10 


Reserved | | | | 


Reading Valid 
Resolution in Deg. Celsius 
Digital Readout 
Cross-domain Limit Log 
Cross-domain Limit Status 
Current Limit Log 
Current Limit Status 
Power Limit Notification Log 
Power Limit Notification Status 
Thermal Threshold #2 Log 
Thermal Threshold #2 Status 
Thermal Threshold #1 Log 
Thermal Threshold #1 Status 
Critical Temperature Log 
Critical Temperature Status 
PROCHOT# or FORCEPR# Log 
PROCHOT# or FORCEPR# Event 
Thermal Status Log 
Thermal Status 


Figure 14-10. [A32_THERM_STATUS Register With HWP Feedback 


Bits 11:0, See Section 14.8.5.2. 


Current Limit Status (bit 12, RO) — If set (1), indicates an electrical current limit (e.g. Electrical Design 
Point/IccMax) is being exceeded and is adversely impacting energy efficiency optimizations. 

Current Limit Log (bit 13, RWCO) — If set (1), an electrical current limit has been exceeded that has 
adversely impacted energy efficiency optimizations since the last clearing of this bit or a reset. This bit is sticky, 
software may clear this bit by writing a zero (0). 

Cross-domain Limit Status (bit 14, RO) — If set (1), indicates another hardware domain (e.g. processor 
graphics) is currently limiting energy efficiency optimizations in the processor core domain. 

Cross-domain Limit Log (bit 15, RWCO) — If set (1), indicates another hardware domain (e.g. processor 
graphics) has limited energy efficiency optimizations in the processor core domain since the last clearing of this 
bit or a reset. This bit is sticky, software may clear this bit by writing a zero (0). 


Bits 63:16, See Section 14.8.5.2. 


14.4.5.1  Non-Architectural HWP Feedback 


The Productive Performance (MSR_PPERF) MSR (non-architectural) provides hardware's view of workload scal- 
ability, which is a rough assessment of the relationship between frequency and workload performance, to software. 
The layout of the MSR_PPERF is shown in Figure 14-11. 


63 0 


PCNT - Productive Performance Count 


Figure 14-11. MSR_PPERF MSR 


PCNT (bits 63:0, RO) — Similar to IA32_APERF but only counts cycles perceived by hardware as contributing 
to instruction execution (e.g. unhalted and unstalled cycles). This counter increments at the same rate as 
IA32_APERF, where the ratio of (APCNT/AACNT) is an indicator of workload scalability (0% to 100%). Note that 
values in this register are valid even when HWP is not enabled. 
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14.4.6 HWP Notifications 


Processors may support interrupt-based notification of changes to HWP status as indicated by CPUID. If supported, 
the IA32_HWP_INTERRUPT MSR is used to enable interrupt-based notifications. Notification events, when enabled, 
are delivered using the existing thermal LVT entry. The layout of the IA32_HWP_INTERRUPT is shown in 

Figure 14-12. The bit fields are described below: 


63 43210 


Reserved 


EN_PECI_OVERRIDE 


EN_Highest_Change 


EN_Excursion_Minimum 


EN_Guaranteed_Performance_Change 


Figure 14-12. IA32_HWP_INTERRUPT MSR 


¢ EN_Guaranteed_Performance_Change (bit 0, RW) — When set (1), an HWP Interrupt will be generated 
whenever a change to the IA32_ HWP_CAPABILITIES.Guaranteed_Performance occurs. The default value is 0 
(Interrupt generation is disabled). 

¢ EN_Excursion_Minimum (bit 1, RW) — When set (1), an HWP Interrupt will be generated whenever the 
HWP hardware is unable to meet the IA32_HWP_REQUEST.Minimum_Performance setting. The default value is 
0 (Interrupt generation is disabled). 


¢ EN_Highest_Change (bit 2, RW) — When set (1), an HWP Interrupt will be generated whenever a change to 
the IA32_HWP_CAPABILITIES.Highest_Performance occurs. The default value is 0 (interrupt generation is 
disabled). Interrupts upon Highest Performance change are supported if CPUID[6].EAX[15] is set. 


¢ EN_PECI_OVERRIDE (bit 3, RW) — When set (1), an HWP Interrupt will be generated whenever PECI starts 
or stops overriding any of the three HWP fields described in Section 14.4.4.3. The default value is 0 (interrupt 
generation is disabled). See Section 14.4.5 and Section 14.4.4.3 for details on how the OS learns what is the 
current set of HWP fields that are overridden by PECI. Interrupts upon PECI override change are supported if 
CPUID[6].EAX[16] is set. 


¢ Bits 63:4 are reserved and must be zero. 


14.4.7. Idle Logical Processor Impact on Core Frequency 
Intel processors use one of two schemes for setting core frequency: 
1. All cores share same frequency. 

2. Each physical core is set to a frequency of its own. 


In both cases the two logical processors that share a single physical core are set to the same frequency, so the 
processor accounts for the IA32_HWP_REQUEST MSR fields of both logical processors when defining the core 
frequency or the whole package frequency. 


When CPUID[6].EAX[20] is set and only one logical processor of the two is active, while the other is idle (in any 
Ci sub-state or in a deeper sleep state), only the active logical processor's IA32_HWP_REQUEST MSR fields 
are considered, i.e., the HWP Request fields of a logical processor in the C1E sub-state or in a deeper sleep state 
are ignored. 


Note: when a logical processor is in C1 state its HWP Request fields are accounted for. 
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14.4.8 Fast Write of Uncore MSR (Model Specific Feature) 


There are a few logical processor scope MSRs whose values need to be observed outside the logical processor. The 
WRMSR instruction takes over 1000 cycles to complete (retire) for those MSRs. This overhead forces operating 
systems to avoid writing them too often whereas in many cases it is preferable that the OS writes them quite 
frequently for optimal power/performance operation of the processor. 


The model specific “Fast Write MSR” feature reduces this overhead by an order of magnitude to a level of 100 cycles 
for a selected subset of MSRs. 


Note: Writes to Fast Write MSRs are posted, i.e., when the WRMSR instruction completes, the data may still be “in 
transit” within the processor. Software can check the status by querying the processor to ensure data is already 
visible outside the logical processor (see Section 14.4.8.3 for additional details). Once the data is visible outside 
the logical processor, software is ensured that later writes by the same logical processor to the same MSR will be 
visible later (will not bypass the earlier writes). 


MSRs that are selected for Fast Write are specified in a special capability MSR (see Section 14.4.8.1). Architectural 
MSRs that existed prior to the introduction of this feature and are selected for Fast Write, thus turning from slow to 
fast write MSRs, will be noted as such via a new CPUID bit. New MSRs that are fast upon introduction will be docu- 
mented as such without an additional CPUID bit. 


Three model specific MSRs are associated with the feature itself. They enable enumerating, controlling and moni- 
toring it. All three are logical processor scope. 


14.4.8.1 FAST_UNCORE_MSRS_CAPABILITY (Address: Ox65F, Logical Processor Scope) 


Operating systems or BIOS can read the FAST_UNCORE_MSRS_CAPABILITY MSR to enumerate those MSRs that 
are Fast Write MSRs. 


63 10 


Reserved 


FAST_IA32_HWP_REQUEST MSR 


Figure 14-13. FAST_UNCORE_MSRS_CAPABILITY MSR 


° FAST_IA32_HWP_REQUEST MSR (bit 0, RO) — When set (1), indicates that the IA32_HWP_REQUEST MSR 
is supported as a Fast Write MSR. A value of 0 indicates the IA32_HWP_REQUEST MSR is not supported as a 
Fast Write MSR. 


¢ Bits 63:1 are reserved and must be zero. 


14.4.8.2 FAST_UNCORE_MSRS_CTL (Address: 0x657, Logical Processor Scope) 


Operating Systems or BIOS can use the FAST_UNCORE_MSRS_CTL MSR to opt-in or opt-out for fast write of 
specific MSRs that are enabled for Fast Write by the processor. 


Note: Not all MSRs that are selected for this feature will necessarily have this opt-in/opt-out option. They may be 
supported in fast write mode only. 
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63 10 


Reserved 


FAST_IA32_HWP_REQUEST_MSR_ENABLE 


Figure 14-14. FAST_UNCORE_MSRS_CTL MSR 


° FAST_IA32_HWP_REQUEST_MSR_ENABLE (bit 0, RW) — When set (1), enables fast access mode for the 
IA32_HWP_REQUEST MSR and sets the low latency, posted IA32_ HWP_REQUESRT MSR' CPUID[6].EAX[18]. 
The default value is 0. Note that this bit can only be enabled once from the default value. Once set, writes to 
this bit are ignored. Only RESET will clear this bit. 


¢ Bits 63:1 are reserved and must be zero. 


14.4.8.3. FAST_UNCORE_MSRS_STATUS (Address: 0x65€, Logical Processor Scope) 


Software that executes the WRMSR instruction of a Fast Write MSR can check whether the data is already visible 
outside the logical processor by reading the FAST_UNCORE_MSRS_STATUS MSR. For each Fast Write MSR there is 
a status bit that indicates whether the data is already visible outside the logical processor or is still in “transit”. 


63 10 


Reserved 


FAST_IA32_HWP_REQUEST_WRITE_STATUS 


Figure 14-15. FAST_UNCORE_MSRS_STATUS MSR 


° FAST_IA32_HWP_REQUEST_WRITE_STATUS (bit 0, RO) — Indicates whether the CPU is still in the 
middle of writing IA32_HWP_REQUEST MSR, even after the WRMSR instruction has retired. A value of 1 
indicates the last write of IA32_HWP_REQUEST is still ongoing. A value of 0 indicates the last write of 
IA32_HWP_REQUEST is visible outside the logical processor. 


¢ Bits 63:1 are reserved and must be zero. 


14.4.9 Fast_IA32_HWP_REQUEST CPUID 


IA32_HWP_REQUEST is an architectural MSR that exists in processors whose CPUID[6].EAX[7] is set (HWP BASE 
is enabled). This MSR has logical processor scope, but after its contents are written the contents become visible 
outside the logical processor. When the FAST_IA32_HWP_REQUEST CPUID[6].EAX[18] bit is set, writes to the 
IA32_HWP_REQUEST MSR are visible outside the logical processor via the “Fast Write” feature described in Section 
14.4.8. 


14.4.10 Recommendations for OS use of HWP Controls 
Common Cases of Using HWP 
The default HWP control field values are expected to be suitable for many applications. The OS can enable autono- 


mous HWP for these common cases by 
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° Setting IA32_HWP_REQUEST. Desired Performance = 0 (hardware autonomous selection determines the 
performance target). Set IA32_HWP_REQUEST.Activity Window = 0 (enable HW dynamic selection of window 
size). 

To maximize HWP benefit for the common cases, the OS should set 

® JTA32_HWP_REQUEST.Minimum_Performance = IA32_HWP_CAPABILITIES.Lowest_Performance and 

¢ = TA32_HWP_REQUEST.Maximum_Performance = IA32_HWP_CAPABILITIES.Highest_Performance. 


Setting IA32_HWP_REQUEST.Minimum_Performance = IA32_HWP_REQUEST.Maximum_Performance is function- 
ally equivalent to using of the IA32_PERF_CTL interface and is therefore not recommended (bypassing HWP). 


Calibrating HWP for Application-Specific HWP Optimization 


In some applications, the OS may have Quality of Service requirements that may not be met by the default values. 
The OS can characterize HWP by: 


° keeping IA32_HWP_REQUEST.Minimum_Performance = IA32_HWP_REQUEST.Maximum_Performance to 
prevent non-linearity in the characterization process, 


¢ utilizing the range values enumerated from the IA32_HWP_CAPABILITIES MSR to program 
IA32_HWP_REQUEST while executing workloads of interest and observing the power and performance result. 


The power and performance result of characterization is also influenced by the IA32_HWP_REQUEST.Energy 
Performance Preference field, which must also be characterized. 


Characterization can be used to set IA32_ HWP_REQUEST.Minimum_Performance to achieve the required QOS in 
terms of performance. If IA32_HWP_REQUEST.Minimum_Performance is set higher than 
IA32_HWP_CAPABILITIES.Guaranteed Performance then notification of excursions to Minimum Performance may 
be continuous. 


If autonomous selection does not deliver the required workload performance, the OS should assess the current 
delivered effective frequency and for the duration of the specific performance requirement set 
IA32_HWP_REQUEST. Desired_Performance # 0 and adjust IA32_HWP_REQUEST.Energy_Performance_Preference 
as necessary to achieve the required workload performance. The MSR_PPERF.PCNT value can be used to better 
comprehend the potential performance result from adjustments to IA32_HWP_REQUEST.Desired_Performance. 
The OS should set IA32_HWP_REQUEST.Desired_Performance = 0 to re-enable autonomous selection. 


Tuning for Maximum Performance or Lowest Power Consumption 


Maximum performance will be delivered by setting IA32_HWP_REQUEST.Minimum_Performance = 
IA32_HWP_REQUEST.Maximum_Performance = IA32_ HWP_CAPABILITIES.Highest_Performance and setting 
IA32_HWP_REQUEST.Energy_Performance_Preference = 0 (performance preference). 


Lowest power will be achieved by setting IA32_ HWP_REQUEST.Minimum_Performance = 
IA32_HWP_REQUEST.Maximum_Performance = IA32_HWP_CAPABILITIES.Lowest_Performance and setting 
IA32_HWP_REQUEST.Energy_Performance_Preference = OFFH (energy efficiency preference). 


Mixing Logical Processor and Package Level HWP Field Settings 


Using the IA32_HWP_REQUEST Package_Control bit and the five valid bits in that MSR, the OS can mix and match 
between selecting the Logical Processor scope fields and the Package level fields. For example, the OS can set all 
logical cores' IA32_ HWP_REQUEST.Package_Control bit to ‘1’, and for those logical processors if it prefers a 
different EPP value than the one set in the IA32_ HWP_REQUEST_PKG MSR, the OS can set the desired EPP value 
and the EPP valid bit. This overrides the package EPP value for only a subset of the logical processors in the 
package. 


Additional Guidelines 


Set IA32_HWP_REQUEST.Energy_Performance_Preference as appropriate for the platform's current mode of oper- 
ation. For example, a mobile platforms’ setting may be towards performance preference when on AC power and 
more towards energy efficiency when on DC power. 
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The use of the Running Average Power Limit (RAPL) processor capability (see section 14.7.1) is highly recom- 
mended when HWP is enabled. Use of IA32_ HWP_Request.Maximum_Performance for thermal control is subject to 
limitations and can adversely impact the performance of other processor components e.g. Graphics 


If default values deliver undesirable performance latency in response to events, the OS should set 
IA32_HWP_REQUEST. Activity_Window to a low (non-zero) value and 
IA32_HWP_REQUEST.Energy_Performance_Preference towards performance (0) for the event duration. 


Similarly, for “real-time” threads, set IA32_HWP_REQUEST.Energy_Performance_Preference towards performance 
(0) and IA32_HWP_REQUEST. Activity_Window to a low value, e.g. 01H, for the duration of their execution. 


When executing low priority work that may otherwise cause the hardware to deliver high performance, set 
IA32_HWP_REQUEST. Activity_Window to a longer value and reduce the 
IA32_HWP_Request.Maximum_Performance value as appropriate to control energy efficiency. Adjustments to 
IA32_HWP_REQUEST.Energy_Performance_Preference may also be necessary. 


14.5 HARDWARE DUTY CYCLING (HDC) 


Intel processors may contain support for Hardware Duty Cycling (HDC), which enables the processor to autono- 
mously force its components inside the physical package into idle state. For example, the processor may selectively 
force only the processor cores into an idle state. 


HDC is disabled by default on processors that support it. System software can dynamically enable or disable HDC 
to force one or more components into an idle state or wake up those components previously forced into an idle 
state. Forced Idling (and waking up) of multiple components in a physical package can be done with one WRMSR to 
a packaged-scope MSR from any logical processor within the same package. 


HDC does not delay events such as timer expiration, but it may affect the latency of short (less than 1 msec) soft- 
ware threads, e.g. if a thread is forced to idle state just before completion and entering a “natural idle”. 


HDC forced idle operation can be thought of as operating at a lower effective frequency. The effective average 
frequency computed by software will include the impact of HDC forced idle. 


The primary use of HDC is enable system software to manage low active workloads to increase the package level 
C6 residency. Additionally, HDC can lower the effective average frequency in case or power or thermal limitation. 


When HDC forces a logical processor, a processor core or a physical package to enter an idle state, its C-State is set 
to C3 or deeper. The deep “C-states” referred to in this section are processor-specific C-states. 


14.5.1 Hardware Duty Cycling Programming Interfaces 


The programming interfaces provided by HDC include the following: 


¢ The CPUID instruction allows software to discover the presence of HDC support in an Intel processor. Specifi- 
cally, execute CPUID instruction with EAX=O6H as input, bit 13 of EAX indicates the processor's support of the 
following aspects of HDC. 


— Availability of HDC baseline resource, CPUID.O6H:EAX[bit 13]: If this bit is set, HDC provides the following 
architectural MSRs: IA32_PKG_HDC_CTL, IA32_PM_CTL1, and the IA32_THREAD_STALL MSRs. 


e =Additionally, HDC may provide several non-architectural MSR. 
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Table 14-3. Architectural and non-Architecture MSRs Related to HDC 


Address | Architec Register Name Description 
tural 

DBOH Y IA32_PKG_HDC_CTL Package Enable/Disable HDC. 

DB1H Y IA32_PM_CTL1 Per-logical-processor select control to allow/block HDC forced idling. 

DB2H Y IA32_THREAD_STALL Accumulate stalled cycles on this logical processor due to HDC forced idling. 

653H N MSR_CORE_HDC_RESIDENCY | Core level stalled cycle counter due to HDC forced idling on one or more 
logical processor. 

655H N MSR_PKG_HDC_SHALLOW_RE | Accumulate the cycles the package was in C2! state and at least one logical 

SIDENCY processor was in forced idle 
656H N MSR_PKG_HDC_DEEP_RESIDE | Accumulate the cycles the package was in the software specified Cx! state 
NCY and at least one logical processor was in forced idle. Cx is specified in 

MSR_PKG_HDC_CONFIG_CTL. 

652H N MSR_PKG_HDC_CONFIG_CTL | HDC configuration controls 

NOTES: 


1. The package “C-states” referred to in this section are processor-specific C-states. 


14.5.2 Package level Enabling HDC 


The layout of the IA32_PKG_HDC_CTL MSR is shown in Figure 14-16. IA32_PKG_HDC_CTLis a writable MSR from 
any logical processor in a package. The bit fields are described below: 


63 10 


Reserved 


Reserved HDC_PKG_Enable 


Figure 14-16. |IA32_PKG_HDC_CTL MSR 


¢ HDC_PKG_Enable (bit 0, R/W) — Software sets this bit to enable HDC operation by allowing the processor 
to force to idle all “HDC-allowed” (see Figure 14.5.3) logical processors in the package. Clearing this bit 
disables HDC operation in the package by waking up all the processor cores that were forced into idle by a 
previous ‘0’-to-’1’ transition in IA32_PKG_HDC_CTL.HDC_PKG_Enable. This bit is writable only if 
CPUID.06H:EAX[bit 13] = 1. Default = zero (0). 


¢ Bits 63:1 are reserved and must be zero. 


After processor support is determined via CPUID, system software can enable HDC operation by setting 
IA32_PKG_HDC_CTL.HDC_PKG_Enable to 1. At reset, IA32_PKG_HDC_CTL.HDC_PKG_Enable is cleared to 0. A 
'O'-to-'1' transition in HDC_PKG_Enable allows the processor to force to idle all HDC-allowed (indicated by the non- 
zero state of IA32_PM_CTL1[bit 0]) logical processors in the package. A ‘1’-to-’0’ transition wakes up those HDC 
force-idled logical processors. 


Software can enable or disable HDC using this package level control multiple times from any logical processor in 
the package. Note the latency of writing a value to the package-visible IA32_PKG_HDC_CTL.HDC_PKG_Enable is 
longer than the latency of a WRMSR operation to a Logical Processor MSR (as opposed to package level MSR) such 
as: IA32_PM_CTLi (described in Section 14.5.3). Propagation of the change in 
IA32_PKG_HDC_CTL.HDC_PKG_Enable and reaching all HDC idled logical processor to be woken up may take on 
the order of core C6 exit latency. 


Vol. 3B 14-21 


POWER AND THERMAL MANAGEMENT 


14.5.3. Logical-Processor Level HDC Control 


The layout of the IA32_PM_CTL1 MSR is shown in Figure 14-17. Each logical processor in a package has its own 
IA32_PM_CTL1 MSR. The bit fields are described below: 


63 10 


Reserved 


HDC_Allow_Block 


Reserved 


Figure 14-17. IA32_PM_CTL1 MSR 


° HDC_Allow_Block (bit 0, R/W) — Software sets this bit to allow this logical processors to honor the 
package-level IA32_PKG_HDC_CTL.HDC_PKG_Enable control. Clearing this bit prevents this logical processor 
from using the HDC. This bit is writable only if CPUID.O6H:EAX[bit 13] = 1. Default = one (1). 


¢ Bits 63:1 are reserved and must be zero. 


Fine-grain OS control of HDC operation at the granularity of per-logical-processor is provided by IA32_PM_CTL1. At 
RESET, all logical processors are allowed to participate in HDC operation such that OS can manage HDC using the 
package-level IA32_PKG_HDC_CTL. 


Writes to IA32_PM_CTL1 complete with the latency that is typical to WRMSR to a Logical Processor level MSR. 
When the OS chooses to manage HDC operation at per-logical-processor granularity, it can write to IA32_PM_CTL1 
on one or more logical processors as desired. Each write to IA32_PM_CTL1 must be done by code that executes on 
the logical processor targeted to be allowed into or blocked from HDC operation. 


Blocking one logical processor for HDC operation may have package level impact. For example, the processor may 
decide to stop duty cycling of all other Logical Processors as well. 


The propagation of IA32_PKG_HDC_CTL.HDC_PKG_Enable in a package takes longer than a WRMSR to 
IA32_PM_CTL1. The last completed write to IA32_PM_CTL1 on a logical processor will be honored when a ‘0’-to-’1’ 
transition of IA32_PKG_HDC_CTL.HDC_PKG_Enable arrives to a logical processor. 


14.5.4 HDC Residency Counters 


There is a collection of counters available for software to track various residency metrics related to HDC operation. 
In general, HDC residency time is defined as the time in HDC forced idle state at the granularity of per-logical- 
processor, per-core, or package. At the granularity of per-core/package-level HDC residency, at least one of the 
logical processor in a core/package must be in the HDC forced idle state. 


14.5.4.1 IA32_THREAD_STALL 


Software can track per-logical-processor HDC residency using the architectural MSR IA32_THREAD_STALL.The 
layout of the IA32_THREAD_STALL MSR is shown in Figure 14-18. Each logical processor in a package has its own 
IA32_THREAD_STALL MSR. The bit fields are described below: 


63 0 


Stall_cycle_cnt 


Figure 14-18. IA32_THREAD_STALL MSR 
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¢ Stall_Cycle_Cnt (bits 63:0, R/O) — Stores accumulated HDC forced-idle cycle count of this processor core 
since last RESET. This counter increments at the same rate of the TSC. The count is updated only after the 
logical processor exits from the forced idled C-state. At each update, the number of cycles that the logical 
processor was Stalled due to forced-idle will be added to the counter. This counter is available only if 
CPUID.06H:EAX[bit 13] = 1. Default = zero (0). 


A value of zero in IA32_THREAD_STALL indicates either HDC is not supported or the logical processor never 
serviced any forced HDC idle. A non-zero value in IA32_THREAD_STALL indicates the HDC forced-idle residency 
times of the logical processor. It also indicates the forced-idle cycles due to HDC that could appear as CO time to 
traditional OS accounting mechanisms (e.g. time-stamping OS idle/exit events). 


Software can read IA32_THREAD_STALL irrespective of the state of IA32_PKG_HDC_CTL and IA32_PM_CTL1, as 
long as CPUID.O6H:EAX[bit 13] = 1. 


14.5.4.2 Non-Architectural HDC Residency Counters 


Processors that support HDC operation may provide the following model-specific HDC residency counters. 


MSR_CORE_HDC_RESIDENCY 


Software can track per-core HDC residency using the counter MSR_CORE_HDC_RESIDENCY. This counter incre- 
ments when the core is in C3 state or deeper (all logical processors in this core are idle due to either HDC or other 
mechanisms) and at least one of the logical processors is in HDC forced idle state. The layout of the 
MSR_CORE_HDC_RESIDENCY is shown in Figure 14-19. Each processor core in a package has its own 
MSR_CORE_HDC_RESIDENCY MSR. The bit fields are described below: 


63 0 


Core_Cx_duty_cycle_cnt 


Figure 14-19. MSR_CORE_HDC_RESIDENCY MSR 


° Core_Cx_Duty_Cycle_Cnt (bits 63:0, R/O) — Stores accumulated HDC forced-idle cycle count of this 
processor core since last RESET. This counter increments at the same rate of the TSC. The count is updated 
only after core C-state exit from a forced idled C-state. At each update, the increment counts cycles when the 
core is in a Cx state (all its logical processor are idle) and at least one logical processor in this core was forced 
into idle state due to HDC. If CPUID.O6H:EAX[bit 13] = 0, attempt to access this MSR will cause a #GP fault. 
Default = zero (0). 


A value of zero in MSR_CORE_HDC_RESIDENCY indicates either HDC is not supported or this processor core never 
serviced any forced HDC idle. 


MSR_PKG_HDC_SHALLOW_RESIDENCY 


The counter MSR_PKG_HDC_SHALLOW_RESIDENCY allows software to track HDC residency time when the 
package is in C2 state, all processor cores in the package are not active and at least one logical processor was 
forced into idle state due to HDC. The layout of the MSR_PKG_HDC_SHALLOW_RESIDENCY is shown in 

Figure 14-20. There is one MSR_PKG_HDC_SHALLOW_RESIDENCY per package. The bit fields are described 
below: 


63 0 


Pkg_Duty_cycle_cnt 


Figure 14-20. MSR_PKG_HDC_SHALLOW_RESIDENCY MSR 


Vol. 3B 14-23 


POWER AND THERMAL MANAGEMENT 


¢ Pkg_Duty_Cycle_Cnt (bits 63:0, R/O) — Stores accumulated HDC forced-idle cycle count of this processor 
core since last RESET. This counter increments at the same rate of the TSC. Package shallow residency may be 
implementation specific. In the initial implementation, the threshold is package C2-state. The count is updated 
only after package C2-state exit from a forced idled C-state. At each update, the increment counts cycles when 
the package is in C2 state and at least one processor core in this package was forced into idle state due to HDC. 
If CPUID.O6H:EAX[bit 13] = 0, attempt to access this MSR may cause a #GP fault. Default = zero (0). 


A value of zero in MSR_PKG_HDC_SHALLOW_RESIDENCY indicates either HDC is not supported or this processor 
package never serviced any forced HDC idle. 


MSR_PKG_HDC_DEEP_RESIDENCY 


The counter MSR_PKG_HDC_DEEP_RESIDENCY allows software to track HDC residency time when the package is 
in a software-specified package Cx state, all processor cores in the package are not active and at least one logical 
processor was forced into idle state due to HDC. Selection of a specific package Cx state can be configured using 
MSR_PKG_HDC_CONFIG. The layout of the MSR_PKG_HDC_DEEP_RESIDENCY is shown in Figure 14-21. There is 
one MSR_PKG_HDC_DEEP_RESIDENCY per package. The bit fields are described below: 


63 0 


Pkg_Cx_duty_cycle_cnt 


Figure 14-21. MSR_PKG_HDC_DEEP_RESIDENCY MSR 


¢ Pkg_Cx_Duty_Cycle_Cnt (bits 63:0, R/O) — Stores accumulated HDC forced-idle cycle count of this 
processor core since last RESET. This counter increments at the same rate of the TSC. The count is updated only 
after package C-state exit from a forced idle state. At each update, the increment counts cycles when the 
package is in the software-configured Cx state and at least one processor core in this package was forced into 
idle state due to HDC. If CPUID.O6H:EAX[bit 13] = 0, attempt to access this MSR may cause a #GP fault. 
Default = zero (0). 


A value of zero in MSR_PKG_HDC_SHALLOW_RESIDENCY indicates either HDC is not supported or this processor 
package never serviced any forced HDC idle. 


MSR_PKG_HDC_CONFIG 


MSR_PKG_HDC_CONFIG allows software to configure the package Cx state that the counter 
MSR_PKG_HDC_DEEP_RESIDENCY monitors. The layout of the MSR_PKG_HDC_CONFIG is shown in Figure 14-22. 
There is one MSR_PKG_HDC_CONFIG per package. The bit fields are described below: 


63 2 0 


Reserved 


HDC_Cx_Monitor 


Reserved 


Figure 14-22. MSR_PKG_HDC_CONFIG MSR 


° Pkg_Cx_Monitor (bits 2:0, R/W) — Selects which package C-state the MSR_HDC_DEEP_RESIDENCY 
counter will monitor. The encoding of the HDC_Cx_Monitor field are: 0: no-counting; 1: count package C2 only, 
2: count package C3 and deeper; 3: count package C6 and deeper; 4: count package C7 and deeper; other 
encodings are reserved. If CPUID.06H:EAX[bit 13] = 0, attempt to access this MSR may cause a #GP fault. 
Default = zero (0). 


¢ Bits 63:3 are reserved and must be zero. 
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14.5.5 MPERF and APERF Counters Under HDC 


HDC operation can be thought of as an average effective frequency drop due to all or some of the Logical Proces- 
sors enter an idle state period. 


1600 MHz: 25% Utilization /75% Forced Idle 


Jo oLI LI LI LI 


Effective Frequency @ 100% Utilization: 400 MHz 


Figure 14-23. Example of Effective Frequency Reduction and Forced Idle Period of HDC 


By default, the IA32_MPERF counter counts during forced idle periods as if the logical processor was active. The 
IA32_APERF counter does not count during forced idle state. This counting convention allows the OS to compute 
the average effective frequency of the Logical Processor between the last MWAIT exit and the next MWAIT entry 
(OS visible CO) by AACNT/AMCNT * TSC Frequency. 


14.6 HARDWARE FEEDBACK INTERFACE 


Intel processors that enumerate CPUID.06H.OH:EAX.HW_FEEDBACK[bit 19] as 1 support Hardware Feedback 
Interface. Hardware provides guidance to the Operating System (OS) scheduler to perform optimal workload 
scheduling through a hardware feedback interface structure in memory. This structure has a global header that is 
16 byte in size. Following this global header, there is one 8 byte entry per logical processor in the socket. The struc- 
ture is designed as follows. 


Table 14-4. Hardware Feedback Interface Structure 


Byte Offset Size (Bytes) Description 

0 16 Global Header 

16 8 Per Logical Processor Entry 
24 8 Per Logical Processor Entry 
16+n*8 8 Per Logical Processor Entry 


The global header is structured as shown in Table 14-5. 
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Table 14-5. Hardware Feedback Interface Global Header Structure 


Byte Offset Size (Bytes) Field Name Description 
0 8 Timestamp Timestamp of when the table was last updated by hardware. This is a 
timestamp in crystal clock units. 
Initialized by OS to O. 
8 1 Performance If set to 1, indicates the performance capability field for one or more logical 
Capability Changed | processors was updated in the table. 
Initialized by OS to O. 
9 1 Energy Efficiency If set to 1, indicates the energy efficiency capability field for one or more 
Capability Changed | logical processors was updated in the table. 
Initialized by OS to O. 
10 6 Reserved Initialized by OS to 0. 


The per logical processor scheduler feedback entry is structured as follows. The operating system can determine 
the index of the logical processor feedback entry for a logical processor using CPUID.06H.OH:EDX[31:16] by 


executing CPUID on that logical processor. 


Table 14-6. Hardware Feedback Interface Logical Processor Entry Structure 


Byte Offset 


Size (Bytes) 


Field Name 


Description 


0 


1 


Performance 
Capability 


Performance capability is an 8-bit value (0 ... 255) specifying the relative 
performance level of a logical processor. Higher values indicate higher perfor- 
mance; the lowest performance level of 0 indicates a recommendation to the OS to 
not schedule any software threads on it for performance reasons. OS scheduler is 
expected to initialize the Hardware Feedback Interface Structure to O prior to 
enabling Hardware Feedback. 

CPUID.O6H.OH:EDX[0] enumerates support for Performance capability reporting. 


Energy 
Efficiency 
Capability 


Energy Efficiency capability is an 8-bit value (0 ... 255) specifying the relative 
energy efficiency level of a logical processor. Higher values indicate higher energy 
efficiency; the lowest energy efficiency capability of O indicates a recommendation 
to the OS to not schedule any software threads on it for efficiency reasons. OS 
scheduler is expected to initialize the Hardware Feedback Interface Structure to 0 
prior to enabling Hardware Feedback. 

CPUID.O6H.OH:EDX[1] enumerates support for Energy Efficiency capability 
reporting. 


14.6.1 


Reserved 


OS scheduler is expected to initialize the Hardware Feedback Interface Structure 
to O prior to enabling Hardware Feedback. 


Hardware Feedback Interface Pointer 


The physical address of the hardware feedback interface structure is programmed by the OS into a package scoped 
MSR named IA32_HW_FEEDBACK_PTR. The MSR is structured as follows: 


* Bits 63:MAXPHYADDR! - Reserved. 
¢ Bits MAXPHYADDR-1:12 - ADDR. This is the physical address of the page frame of the first page of this 


structure. 


e = ©Bits 11:1 - 


Reserved. 


° Bit O - Valid. When set to 1, indicates a valid pointer is programmed into the ADDR field of the MSR. 
The address of this MSR is 17D0H. 


1. MAXPHYADDR is reported in CPUID.B0000008H:EAX[7:0]. 
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CPUID.06H.OH:EDX[11:8] enumerates the size of memory that must be allocated by the OS for this structure. 


14.6.2 | Hardware Feedback Interface Configuration 


The operating system enables the hardware feedback interface using a package scoped MSR named 
IA32_HW_FEEDBACK_CONFIG (address 17D1H). 


The MSR is structured as follows: 
° = Bits 63:1 - Reserved. 
¢ Bit 0 - Enable. When set to 1, enables the hardware feedback interface. 


Before enabling the hardware feedback interface, the OS must set a valid hardware feedback interface structure 
using IA32_HW_FEEDBACK_PTR. 


When the Enable bit transitions from 1 to 0, hardware sets the IA32_PACKAGE_THERM_STATUS bit 26 to 1 to 
acknowledge disabling of the interface. The OS must wait for this bit to be set to 1 after disabling the interface 
before reclaiming the memory allocated for this structure. When this bit is set to 1, it is safe to reclaim the memory 
as it is guaranteed that there are no writes in progress to this structure by hardware. 


Execution of GETSEC[SENTER] clears the enable bit to 0 on all sockets in the platform. 


14.6.3 Hardware Feedback Interface Notifications 


The IA32_PACKAGE_THERM_STATUS MSR is extended with a new bit, hardware feedback interface structure 
change status (bit 26, R/WCO), to indicate that the hardware has updated the hardware feedback interface struc- 
ture. This is a sticky bit and once set, indicates that the OS should read the structure to determine the change and 
adjust its scheduling decisions. Once set, the hardware will not generate any further updates to this structure until 
the OS clears this bit by writing 0. 


The OS can enable interrupt-based notifications when the structure is updated by hardware through a new enable 
bit, hardware feedback interrupt enable (bit 25, R/W), in the IA32_PACKAGE_THERM_INTERRUPT MSR. When this 
bit is set to 1, it enables the generation of an interrupt when the hardware feedback interface structure is updated 
by hardware. When the Enable bit transitions from 0 to 1, hardware will generate an initial notify, with the 
IA32_PACKAGE_THERM_STATUS bit 26 set to 1, to indicate that the OS should read the current hardware feedback 
interface structure. 


14.7. MWAIT EXTENSIONS FOR ADVANCED POWER MANAGEMENT 


IA-32 processors may support a number of C-states* that reduce power consumption for inactive states. Intel Core 
Solo and Intel Core Duo processors support both deeper C-state and MWAIT extensions that can be used by OS to 
implement power management policy. 


Software should use CPUID to discover if a target processor supports the enumeration of MWAIT extensions. If 
CPUID.0O5H.ECX[Bit 0] = 1, the target processor supports MWAIT extensions and their enumeration (see Chapter 
4, “Instruction Set Reference, M-U,” of Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
2B). 

If CPUID.O5H.ECX[Bit 1] = 1, the target processor supports using interrupts as break-events for MWAIT, even 
when interrupts are disabled. Use this feature to measure C-state residency as follows: 


°* Software can write to bit 0 in the MWAIT Extensions register (ECX) when issuing an MWAIT to enter into a 
processor-specific C-state or sub C-state. 


¢ When a processor comes out of an inactive C-state or sub C-state, software can read a timestamp before an 
interrupt service routine (ISR) is potentially executed. 


2. The processor-specific C-states defined in MWAIT extensions can map to ACPI defined C-state types (CO, C1, C2, C3). The mapping 
relationship depends on the definition of a C-state by processor implementation and is exposed to OSPM by the BIOS using the ACPI 
defined _CST table. 


Vol. 3B 14-27 


POWER AND THERMAL MANAGEMENT 


CPUID.05H.EDX allows software to enumerate processor-specific C-states and sub C-states available for use with 
MWAIT extensions. IA-32 processors may support more than one C-state of a given C-state type. These are called 
sub C-states. Numerically higher C-state have higher power savings and latency (upon entering and exiting) than 
lower-numbered C-state. 


At CPL = 0, system software can specify desired C-state and sub C-state by using the MWAIT hints register (EAX). 
Processors will not go to C-state and sub C-state deeper than what is specified by the hint register. If CPL > 0 and 
if MONITOR/MWATT is supported at CPL > 0, the processor will only enter Ci-state (regardless of the C-state 
request in the hints register). 


Executing MWAIT generates an exception on processors operating at a privilege level where MONITOR/MWAIT are 
not supported. 


NOTE 


If MWAIT is used to enter a C-state (including sub C-state) that is numerically higher than Cl, a 
store to the address range armed by MONITOR instruction will cause the processor to exit MWAIT if 
the store was originated by other processor agents. A store from non-processor agent may not 
cause the processor to exit MWAIT. 


14.8 THERMAL MONITORING AND PROTECTION 


The IA-32 architecture provides the following mechanisms for monitoring temperature and controlling thermal 
power: 


1. The catastrophic shutdown detector forces processor execution to stop if the processor’s core temperature 
rises above a preset limit. 


2. Automatic and adaptive thermal monitoring mechanisms force the processor to reduce it’s power 
consumption in order to operate within predetermined temperature limits. 


3. The software controlled clock modulation mechanism permits operating systems to implement power 
management policies that reduce power consumption; this is in addition to the reduction offered by automatic 
thermal monitoring mechanisms. 


4. On-die digital thermal sensor and interrupt mechanisms permit the OS to manage thermal conditions 
natively without relying on BIOS or other system board components. 


The first mechanism is not visible to software. The other three mechanisms are visible to software using processor 
feature information returned by executing CPUID with EAX = 1. 


The second mechanism includes: 


¢ Automatic thermal monitoring provides two modes of operation. One mode modulates the clock duty cycle; 
the second mode changes the processor’s frequency. Both modes are used to control the core temperature of 
the processor. 


¢ Adaptive thermal monitoring can provide flexible thermal management on processors made of multiple 
cores. 


The third mechanism modulates the clock duty cycle of the processor. As shown in Figure 14-24, the phrase ‘duty 
cycle’ does not refer to the actual duty cycle of the clock signal. Instead it refers to the time period during which the 
clock signal is allowed to drive the processor chip. By using the stop clock mechanism to control how often the 
processor is clocked, processor power consumption can be modulated. 
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Clock Applied to Processor 


Stop-Clock Duty Cycle 


25% Duty Cycle (example only) 


Figure 14-24. Processor Modulation Through Stop-Clock Mechanism 


For previous automatic thermal monitoring mechanisms, software controlled mechanisms that changed processor 
operating parameters to impact changes in thermal conditions. Software did not have native access to the native 
thermal condition of the processor; nor could software alter the trigger condition that initiated software program 

control. 


The fourth mechanism (listed above) provides access to an on-die digital thermal sensor using a model-specific 
register and uses an interrupt mechanism to alert software to initiate digital thermal monitoring. 


14.8.1 Catastrophic Shutdown Detector 


P6 family processors introduced a thermal sensor that acts as a catastrophic shutdown detector. This catastrophic 
shutdown detector was also implemented in Pentium 4, Intel Xeon and Pentium M processors. It is always enabled. 
When processor core temperature reaches a factory preset level, the sensor trips and processor execution is halted 
until after the next reset cycle. 


14.8.2 | Thermal Monitor 


Pentium 4, Intel Xeon and Pentium M processors introduced a second temperature sensor that is factory-calibrated 
to trip when the processor’s core temperature crosses a level corresponding to the recommended thermal design 
envelop. The trip-temperature of the second sensor is calibrated below the temperature assigned to the cata- 
strophic shutdown detector. 


14.8.2.1 Thermal Monitor 1 


The Pentium 4 processor uses the second temperature sensor in conjunction with a mechanism called Thermal 
Monitor 1 (TM1) to control the core temperature of the processor. TM1 controls the processor’s temperature by 
modulating the duty cycle of the processor clock. Modulation of duty cycles is processor model specific. Note that 
the processors STPCLK# pin is not used here; the stop-clock circuitry is controlled internally. 


Support for TM1 is indicated by CPUID.1:EDX.TM[bit 29] = 1. 


TM1 is enabled by setting the thermal-monitor enable flag (bit 3) in IA32_MISC_ENABLE [see Chapter 2, “Model- 
Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4]. 
Following a power-up or reset, the flag is cleared, disabling TM1. BIOS is required to enable only one automatic 
thermal monitoring modes. Operating systems and applications must not disable the operation of these mecha- 
nisms. 


14.8.2.2 Thermal Monitor 2 


An additional automatic thermal protection mechanism, called Thermal Monitor 2 (TM2), was introduced in the 
Intel Pentium M processor and also incorporated in newer models of the Pentium 4 processor family. Intel Core Duo 
and Solo processors, and Intel Core 2 Duo processor family all support TM1 and TM2. TM2 controls the core 
temperature of the processor by reducing the operating frequency and voltage of the processor and offers a higher 
performance level for a given level of power reduction than TM1. 
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TM2 is triggered by the same temperature sensor as TM1. The mechanism to enable TM2 may be implemented 
differently across various IA-32 processor families with different CPUID signatures in the family encoding value, but 
will be uniform within an IA-32 processor family. 


Support for TM2 is indicated by CPUID.1:ECX.TM2[bit 8] = 1. 


14.8.2.3. Two Methods for Enabling TM2 


On processors with CPUID family/model/stepping signature encoded as 0x69n or Ox6Dn (early Pentium M proces- 
sors), TM2 is enabled if the TM_SELECT flag (bit 16) of the MSR_THERM2_CTL register is set to 1 (Figure 14-25) 
and bit 3 of the IA32_MISC_ENABLE register is set to 1. 


Following a power-up or reset, the TM_SELECT flag may be cleared. BIOS is required to enable either TM1 or TM2. 
Operating systems and applications must not disable mechanisms that enable TM1 or TM2. If bit 3 of the 
IA32_MISC_ENABLE register is set and TM_SELECT flag of the MSR_THERM2_CTL register is cleared, TM1 is 
enabled. 


31 16 0 
Reserved Reserved 


TM_SELECT 


Figure 14-25. MSR_THERM2_CTL Register On Processors with CPUID Family/Model/Stepping Signature Encoded 
as Ox69n or Ox6Dn 


On processors introduced after the Pentium 4 processor (this includes most Pentium M processors), the method 
used to enable TM2 is different. TM2 is enable by setting bit 13 of IA32_MISC_ENABLE register to 1. This applies to 
Intel Core Duo, Core Solo, and Intel Core 2 processor family. 


The target operating frequency and voltage for the TM2 transition after TM2 is triggered is specified by the value 
written to MSR_THERM2_CTL, bits 15:0 (Figure 14-26). Following a power-up or reset, BIOS is required to enable 
at least one of these two thermal monitoring mechanisms. If both TM1 and TM2 are supported, BIOS may choose 
to enable TM2 instead of TM1. Operating systems and applications must not disable the mechanisms that enable 
TMior TM2; and they must not alter the value in bits 15:0 of the MSR_THERM2_CTL register. 


Reserved 


TM2 Transition Target | 


Figure 14-26. MSR_THERM2_CTL Register for Supporting TM2 


14.8.2.4 Performance State Transitions and Thermal Monitoring 


If the thermal control circuitry (TCC) for thermal monitor (TM1/TM2) is active, writes to the IA32_PERF_CTL will 
effect a new target operating point as follows: 


° If TM1 is enabled and the TCC is engaged, the performance state transition can commence before the TCC is 
disengaged. 
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¢ If TM2 is enabled and the TCC is engaged, the performance state transition specified by a write to the 
IA32_PERF_CTL will commence after the TCC has disengaged. 


14.8.2.5 Thermal Status Information 


The status of the temperature sensor that triggers the thermal monitor (TM1/TM2) is indicated through the thermal 
status flag and thermal status log flag in the IA32_THERM_STATUS MSR (see Figure 14-27). 


The functions of these flags are: 


° Thermal Status flag, bit 0 — When set, indicates that the processor core temperature is currently at the trip 
temperature of the thermal monitor and that the processor power consumption is being reduced via either TM1 
or TM2, depending on which is enabled. When clear, the flag indicates that the core temperature is below the 
thermal monitor trip temperature. This flag is read only. 


¢ Thermal Status Log flag, bit 1 — When set, indicates that the thermal sensor has tripped since the last 
power-up or reset or since the last time that software cleared this flag. This flag is a sticky bit; once set it 
remains set until cleared by software or until a power-up or reset of the processor. The default state is clear. 


63 210 


Reserved 


Thermal Status Log = 


Thermal Status 


Figure 14-27. IA32_THERM_STATUS MSR 


After the second temperature sensor has been tripped, the thermal monitor (TM1/TM2) will remain engaged for a 
minimum time period (on the order of 1 ms). The thermal monitor will remain engaged until the processor core 
temperature drops below the preset trip temperature of the temperature sensor, taking hysteresis into account. 


While the processor is in a stop-clock state, interrupts will be blocked from interrupting the processor. This holding 
off of interrupts increases the interrupt latency, but does not cause interrupts to be lost. Outstanding interrupts 
remain pending until clock modulation is complete. 


The thermal monitor can be programmed to generate an interrupt to the processor when the thermal sensor is 
tripped; this is called a thermal interrupt. The delivery mode, mask and vector for this interrupt can be 
programmed through the thermal entry in the local APIC’s LVT (see Section 10.5.1, “Local Vector Table”). The low- 
temperature interrupt enable and high-temperature interrupt enable flags in the IA32_THERM_INTERRUPT MSR 
(see Figure 14-28) control when the interrupt is generated; that is, on a transition from a temperature below the 
trip point to above and/or vice-versa. 


63 


Reserved 


210 
Low-Temperature Interrupt Enable __ | 


High-Temperature Interrupt Enable 


Figure 14-28. IA32_THERM_INTERRUPT MSR 


¢ High-Temperature Interrupt Enable flag, bit 0 — Enables an interrupt to be generated on the transition 
from a low-temperature to a high-temperature when set; disables the interrupt when clear.(R/W). 


¢ Low-Temperature Interrupt Enable flag, bit 1 — Enables an interrupt to be generated on the transition 
from a high-temperature to a low-temperature when set; disables the interrupt when clear. 


The thermal interrupt can be masked by the thermal LVT entry. After a power-up or reset, the low-temperature 
interrupt enable and high-temperature interrupt enable flags in the IA32_THERM_INTERRUPT MSR are cleared 
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(interrupts are disabled) and the thermal LVT entry is set to mask interrupts. This interrupt should be handled 
either by the operating system or system management mode (SMM) code. 


Note that the operation of the thermal monitoring mechanism has no effect upon the clock rate of the processor's 
internal high-resolution timer (time stamp counter). 


14.8.2.6 Adaptive Thermal Monitor 


The Intel Core 2 Duo processor family supports enhanced thermal management mechanism, referred to as Adap- 
tive Thermal Monitor (Adaptive TM). 


Unlike TM2, Adaptive TM is not limited to one TM2 transition target. During a thermal trip event, Adaptive TM (if 
enabled) selects an optimal target operating point based on whether or not the current operating point has effec- 
tively cooled the processor. 


Similar to TM2, Adaptive TM is enable by BIOS. The BIOS is required to test the TM1 and TM2 feature flags and 
enable all available thermal control mechanisms (including Adaptive TM) at platform initiation. 


Adaptive TM is available only to a subset of processors that support TM2. 


In each chip-multiprocessing (CMP) silicon die, each core has a unique thermal sensor that triggers independently. 
These thermal sensor can trigger TM1 or TM2 transitions in the same manner as described in Section 14.8.2.1 and 
Section 14.8.2.2. The trip point of the thermal sensor is not programmable by software since it is set during the 
fabrication of the processor. 


Each thermal sensor in a processor core may be triggered independently to engage thermal management features. 
In Adaptive TM, both cores will transition to a lower frequency and/or lower voltage level if one sensor is triggered. 


Triggering of this sensor is visible to software via the thermal interrupt LVT entry in the local APIC of a given core. 


14.8.3. Software Controlled Clock Modulation 


Pentium 4, Intel Xeon and Pentium M processors also support software-controlled clock modulation. This provides 
a means for operating systems to implement a power management policy to reduce the power consumption of the 
processor. Here, the stop-clock duty cycle is controlled by software through the IA32_CLOCK_MODULATION MSR 
(see Figure 14-29). 


63 543 10 


Reserved 


On-Demand Clock Modulation Enable 
On-Demand Clock Modulation Duty Cycle 


Reserved 


Figure 14-29. 1A32_CLOCK_MODULATION MSR 


The IA32_CLOCK_MODULATION MSR contains the following flag and field used to enable software-controlled clock 
modulation and to select the clock modulation duty cycle: 


¢ On-Demand Clock Modulation Enable, bit 4 — Enables on-demand software controlled clock modulation 
when set; disables software-controlled clock modulation when clear. 


* On-Demand Clock Modulation Duty Cycle, bits 1 through 3 — Selects the on-demand clock modulation 
duty cycle (see Table 14-7). This field is only active when the on-demand clock modulation enable flag is set. 


Note that the on-demand clock modulation mechanism (like the thermal monitor) controls the processor’s stop- 
clock circuitry internally to modulate the clock signal. The STPCLK# pin is not used in this mechanism. 
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Table 14-7. On-Demand Clock Modulation Duty Cycle Field Encoding 


Duty Cycle Field Encoding Duty Cycle 
000B Reserved 
001B 12.5% (Default) 
010B 25.0% 
011B 37.5% 
100B 50.0% 
101B 63.5% 
110B 75% 
111B 87.5% 


The on-demand clock modulation mechanism can be used to control processor power consumption. Power 
management software can write to the IA32_CLOCK_MODULATION MSR to enable clock modulation and to select 
a modulation duty cycle. If on-demand clock modulation and TM1 are both enabled and the thermal status of the 
processor is hot (bit 0 of the IA32_THERM_STATUS MSR is set), clock modulation at the duty cycle specified by TM1 
takes precedence, regardless of the setting of the on-demand clock modulation duty cycle. 


For Hyper-Threading Technology enabled processors, the IA32_CLOCK_MODULATION register is duplicated for 
each logical processor. In order for the On-demand clock modulation feature to work properly, the feature must be 
enabled on all the logical processors within a physical processor. If the programmed duty cycle is not identical for 
all the logical processors, the processor core clock will modulate to the highest duty cycle programmed for proces- 
sors with any of the following CPUID DisplayFamily_DisplayModel signatures (see CPUID instruction in Chapter3, 
“Instruction Set Reference, A-L” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
2A): 06_1A, 06_1C, 06_1E, 06_1F, 06_25, 06_26, 06_27, 06_2C, 06_2E, 06_2F, 06_35, 06_36, and OF_xx. For all 
other processors, if the programmed duty cycle is not identical for all logical processors in the same core, the 
processor core will modulate at the lowest programmed duty cycle. 


For multiple processor cores in a physical package, each processor core can modulate to a programmed duty cycle 
independently. 


For the P6 family processors, on-demand clock modulation was implemented through the chipset, which controlled 
clock modulation through the processor’s STPCLK# pin. 


14.8.3.1 Extension of Software Controlled Clock Modulation 


Extension of the software controlled clock modulation facility supports on-demand clock modulation duty cycle with 
4-bit dynamic range (increased from 3-bit range). Granularity of clock modulation duty cycle is increased to 6.25% 
(compared to 12.5%). 


Four bit dynamic range control is provided by using bit 0 in conjunction with bits 3:1 of the 
IA32_CLOCK_MODULATION MSR (see Figure 14-30). 


63 543 0 


Reserved 


On-Demand Clock Modulation Enable 


Extended On-Demand Clock Modulation Duty Cycle ——— 
Reserved 


Figure 14-30. 1A32_CLOCK_MODULATION MSR with Clock Modulation Extension 


Extension to software controlled clock modulation is supported only if CPUID.06H:EAX[Bit 5] = 1. If 
CPUID.06H:EAX[Bit 5] = 0, then bit 0 of IA32_CLOCK_MODULATION is reserved. 
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14.8.4 Detection of Thermal Monitor and Software Controlled Clock Modulation Facilities 


The ACPI flag (bit 22) of the CPUID feature flags indicates the presence of the IA32_THERM_STATUS, 
IA32_THERM_INTERRUPT, IA32_CLOCK_MODULATION MSRs, and the xAPIC thermal LVT entry. 


The TM1 flag (bit 29) of the CPUID feature flags indicates the presence of the automatic thermal monitoring facili- 
ties that modulate clock duty cycles. 


14.8.4.1 Detection of Software Controlled Clock Modulation Extension 


Processor’s support of software controlled clock modulation extension is indicated by CPUID.O6H:EAX[Bit 5] = 1. 


14.8.5 On Die Digital Thermal Sensors 


On die digital thermal sensor can be read using an MSR (no I/O interface). In Intel Core Duo processors, each core 
has a unique digital sensor whose temperature is accessible using an MSR. The digital thermal sensor is the 
preferred method for reading the die temperature because (a) it is located closer to the hottest portions of the die, 
(b) it enables software to accurately track the die temperature and the potential activation of thermal throttling. 


14.8.5.1 Digital Thermal Sensor Enumeration 


The processor supports a digital thermal sensor if CPUID.O6H.EAX[0] = 1. If the processor supports digital thermal 
sensor, EBX[bits 3:0] determine the number of thermal thresholds that are available for use. 


Software sets thermal thresholds by using the IA32_THERM_INTERRUPT MSR. Software reads output of the digital 
thermal sensor using the IA32_THERM_STATUS MSR. 


14.8.5.2 Reading the Digital Sensor 


Unlike traditional analog thermal devices, the output of the digital thermal sensor is a temperature relative to the 
maximum supported operating temperature of the processor. 


Temperature measurements returned by digital thermal sensors are always at or below TCC activation tempera- 
ture. Critical temperature conditions are detected using the “Critical Temperature Status” bit. When this bit is set, 
the processor is operating at a critical temperature and immediate shutdown of the system should occur. Once the 
“Critical Temperature Status” bit is set, reliable operation is not guaranteed. 


See Figure 14-31 for the layout of IA32_THERM_STATUS MSR. Bit fields include: 


° Thermal Status (bit 0, RO) — This bit indicates whether the digital thermal sensor high-temperature output 
signal (PROCHOT#) is currently active. Bit 0 = 1 indicates the feature is active. This bit may not be written by 
software; it reflects the state of the digital thermal sensor. 

* Thermal Status Log (bit 1, R/WCO) — This is a sticky bit that indicates the history of the thermal sensor 
high temperature output signal (PROCHOT#). Bit 1 = 1 if PROCHOT# has been asserted since a previous 
RESET or the last time software cleared the bit. Software may clear this bit by writing a zero. 

¢ PROCHOT# or FORCEPR# Event (bit 2, RO) — Indicates whether PROCHOT# or FORCEPR# is being 
asserted by another agent on the platform. 
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Reserved 


Reading Valid _ 


Resolution in Deg. Celsius 
Digital Readout 
Power Limit Notification Log 
Power Limit Notification Status 
Thermal Threshold #2 Log 
Thermal Threshold #2 Status 
Thermal Threshold #1 Log 

Thermal Threshold #1 Status 
Critical Temperature Log 
Critical Temperature Status 
PROCHOT# or FORCEPR# Log 
PROCHOT# or FORCEPR# Event 
Thermal Status Log 
Thermal Status 


Figure 14-31. IA32_THERM_STATUS Register 


PROCHOT# or FORCEPR# Log (bit 3, R/WCO) — Sticky bit that indicates whether PROCHOT# or 
FORCEPR# has been asserted by another agent on the platform since the last clearing of this bit or a reset. If 
bit 3 = 1, PROCHOT# or FORCEPR# has been externally asserted. Software may clear this bit by writing a zero. 
External PROCHOT# assertions are only acknowledged if the Bidirectional Prochot feature is enabled. 


Critical Temperature Status (bit 4, RO) — Indicates whether the critical temperature detector output signal 
is currently active. If bit 4 = 1, the critical temperature detector output signal is currently active. 


Critical Temperature Log (bit 5, R/WCO) — Sticky bit that indicates whether the critical temperature 
detector output signal has been asserted since the last clearing of this bit or reset. If bit 5 = 1, the output 
signal has been asserted. Software may clear this bit by writing a zero. 


Thermal Threshold #1 Status (bit 6, RO) — Indicates whether the actual temperature is currently higher 
than or equal to the value set in Thermal Threshold #1. If bit 6 = 0, the actual temperature is lower. If 

bit 6 = 1, the actual temperature is greater than or equal to TT#1. Quantitative information of actual 
temperature can be inferred from Digital Readout, bits 22:16. 


Thermal Threshold #1 Log (bit 7, R/ WCO) — Sticky bit that indicates whether the Thermal Threshold #1 
has been reached since the last clearing of this bit or a reset. If bit 7 = 1, the Threshold #1 has been reached. 
Software may clear this bit by writing a zero. 


Thermal Threshold #2 Status (bit 8, RO) — Indicates whether actual temperature is currently higher than 
or equal to the value set in Thermal Threshold #2. If bit 8 = 0, the actual temperature is lower. If bit 8 = 1, the 
actual temperature is greater than or equal to TT#2. Quantitative information of actual temperature can be 
inferred from Digital Readout, bits 22:16. 


Thermal Threshold #2 Log (bit 9, R/WCO) — Sticky bit that indicates whether the Thermal Threshold #2 
has been reached since the last clearing of this bit or a reset. If bit 9 = 1, the Thermal Threshold #2 has been 
reached. Software may clear this bit by writing a zero. 


Power Limitation Status (bit 10, RO) — Indicates whether the processor is currently operating below OS- 
requested P-state (specified in IA32_PERF_CTL) or OS-requested clock modulation duty cycle (specified in 
IA32_CLOCK_MODULATION). This field is supported only if CPUID.O6H:EAX[bit 4] = 1. Package level power 
limit notification can be delivered independently to IA32_PACKAGE_THERM_STATUS MSR. 


Power Notification Log (bit 11, R/WCO) — Sticky bit that indicates the processor went below OS- 
requested P-state or OS-requested clock modulation duty cycle since the last clearing of this or RESET. This 
field is supported only if CPUID.0O6H:EAX[bit 4] = 1. Package level power limit notification is indicated indepen- 
dently in IA32_PACKAGE_THERM_STATUS MSR. 
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Digital Readout (bits 22:16, RO) — Digital temperature reading in 1 degree Celsius relative to the TCC 
activation temperature. 


0: TCC Activation temperature, 
1: (TCC Activation - 1) , etc. See the processor’s data sheet for details regarding TCC activation. 
A lower reading in the Digital Readout field (bits 22:16) indicates a higher actual temperature. 


Resolution in Degrees Celsius (bits 30:27, RO) — Specifies the resolution (or tolerance) of the digital 
thermal sensor. The value is in degrees Celsius. It is recommended that new threshold values be offset from the 
current temperature by at least the resolution + 1 in order to avoid hysteresis of interrupt generation. 


Reading Valid (bit 31, RO) — Indicates if the digital readout in bits 22:16 is valid. The readout is valid if 
bit 31 = 1. 


Changes to temperature can be detected using two thresholds (see Figure 14-32); one is set above and the other 
below the current temperature. These thresholds have the capability of generating interrupts using the core's local 
APIC which software must then service. Note that the local APIC entries used by these thresholds are also used by 
the Intel® Thermal Monitor; it is up to software to determine the source of a specific interrupt. 
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Reserved 


Power Limit Notification sie: 


Threshold #2 Interrupt Enable 
Threshold #2 Value 
Threshold #1 Interrupt Enable 
Threshold #1 Value 
Overheat Interrupt Enable 
FORCPR# Interrupt Enable 
PROCHOT# Interrupt Enable 
Low Temp. Interrupt Enable 
High Temp. Interrupt Enable 


Figure 14-32. |A32_THERM_INTERRUPT Register 


See Figure 14-32 for the layout of IA32_THERM_INTERRUPT MSR. Bit fields include: 


High-Temperature Interrupt Enable (bit 0, R/W) — This bit allows the BIOS to enable the generation of 
an interrupt on the transition from low-temperature to a high-temperature threshold. Bit 0 = 0 (default) 
disables interrupts; bit 0 = 1 enables interrupts. 


Low-Temperature Interrupt Enable (bit 1, R/W) — This bit allows the BIOS to enable the generation of an 
interrupt on the transition from high-temperature to a low-temperature (TCC de-activation). Bit 1 = 0 (default) 
disables interrupts; bit 1 = 1 enables interrupts. 


PROCHOT# Interrupt Enable (bit 2, R/W) — This bit allows the BIOS or OS to enable the generation of an 
interrupt when PROCHOT# has been asserted by another agent on the platform and the Bidirectional Prochot 
feature is enabled. Bit 2 = O disables the interrupt; bit 2 = 1 enables the interrupt. 


FORCEPR# Interrupt Enable (bit 3, R/W) — This bit allows the BIOS or OS to enable the generation of an 
interrupt when FORCEPR# has been asserted by another agent on the platform. Bit 3 = 0 disables the 
interrupt; bit 3 = 1 enables the interrupt. 


Critical Temperature Interrupt Enable (bit 4, R/W) — Enables the generation of an interrupt when the 
Critical Temperature Detector has detected a critical thermal condition. The recommended response to this 
condition is a system shutdown. Bit 4 = 0 disables the interrupt; bit 4 = 1 enables the interrupt. 


Threshold #1 Value (bits 14:8, R/W) — A temperature threshold, encoded relative to the TCC Activation 
temperature (using the same format as the Digital Readout). This threshold is compared against the Digital 
Readout and is used to generate the Thermal Threshold #1 Status and Log bits as well as the Threshold #1 
thermal interrupt delivery. 
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¢ Threshold #1 Interrupt Enable (bit 15, R/W) — Enables the generation of an interrupt when the actual 
temperature crosses the Threshold #1 setting in any direction. Bit 15 = 1 enables the interrupt; bit 15 = 0 
disables the interrupt. 


¢ Threshold #2 Value (bits 22:16, R/W) —A temperature threshold, encoded relative to the TCC Activation 
temperature (using the same format as the Digital Readout). This threshold is compared against the Digital 
Readout and is used to generate the Thermal Threshold #2 Status and Log bits as well as the Threshold #2 
thermal interrupt delivery. 


¢ Threshold #2 Interrupt Enable (bit 23, R/W) — Enables the generation of an interrupt when the actual 
temperature crosses the Threshold #2 setting in any direction. Bit 23 = lenables the interrupt; bit 23 = 0 
disables the interrupt. 

° Power Limit Notification Enable (bit 24, R/W) — Enables the generation of power notification events when 
the processor went below OS-requested P-state or OS-requested clock modulation duty cycle. This field is 
supported only if CPUID.O6H:EAX[bit 4] = 1. Package level power limit notification can be enabled indepen- 
dently by IA32_PACKAGE_THERM_INTERRUPT MSR. 


14.8.6 Power Limit Notification 


Platform firmware may be capable of specifying a power limit to restrict power delivered to a platform component, 
such as a physical processor package. This constraint imposed by platform firmware may occasionally cause the 
processor to operate below OS-requested P or T-state. A power limit notification event can be delivered using the 
existing thermal LVT entry in the local APIC. 


Software can enumerate the presence of the processor’s support for power limit notification by verifying 
CPUID.0O6H:EAX[bit 4] = 1. 

If CPUID.O6H:EAX[bit 4] = 1, then IA32_THERM_INTERRUPT and IA32_THERM_STATUS provides the following 
facility to manage power limit notification: 


¢ = Bits 10 and 11 in IA32_THERM_STATUS informs software of the occurrence of processor operating below OS- 
requested P-state or clock modulation duty cycle setting (see Figure 14-31). 


° Bit 24 in IA32_THERM_INTERRUPT enables the local APIC to deliver a thermal event when the processor went 
below OS-requested P-state or clock modulation duty cycle setting (see Figure 14-32). 


14.9 PACKAGE LEVEL THERMAL MANAGEMENT 


The thermal management facilities like IA32_THERM_INTERRUPT and IA32_THERM_STATUS are often imple- 
mented with a processor core granularity. To facilitate software manage thermal events from a package level gran- 
ularity, two architectural MSR is provided for package level thermal management. The 
IA32_PACKAGE_THERM_STATUS and IA32_PACKAGE_THERM_INTERRUPT MSRs use similar interfaces as 
IA32_THERM_STATUS and IA32_THERM_INTERRUPT, but are shared in each physical processor package. 


Software can enumerate the presence of the processor’s support for package level thermal management facility 
(IA32_PACKAGE_THERM_STATUS and IA32_PACKAGE_THERM_INTERRUPT) by verifying CPUID.O6H:EAX[bit 6] = 
1. 


The layout of IA32_PACKAGE_THERM_STATUS MSR is shown in Figure 14-33. 
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Reserved 


PKG Digital Readout 
PKG Power Limit Notification Log 
PKG Power Limit Notification Status 
PKG Thermal Threshold #2 Log 
PKG Thermal Threshold #2 Status 
PKG Thermal Threshold #1 Log 
PKG Thermal Threshold #1 Status 
PKG Critical Temperature Log 
PKG Critical Temperature Status 
PKG PROCHOT# or FORCEPR# Log 
PKG PROCHOT# or FORCEPR# Event 
PKG Thermal Status Log 
PKG Thermal Status 


Figure 14-33. IA32_PACKAGE_THERM_STATUS Register 


¢ Package Thermal Status (bit 0, RO) — This bit indicates whether the digital thermal sensor high- 
temperature output signal (PROCHOT#) for the package is currently active. Bit 0 = 1 indicates the feature is 
active. This bit may not be written by software; it reflects the state of the digital thermal sensor. 


¢ Package Thermal Status Log (bit 1, R/WCO) — This is a sticky bit that indicates the history of the thermal 
sensor high temperature output signal (PROCHOT#) of the package. Bit 1 = 1 if package PROCHOT# has been 
asserted since a previous RESET or the last time software cleared the bit. Software may clear this bit by writing 
a zero. 


¢ Package PROCHOT# Event (bit 2, RO) — Indicates whether package PROCHOT# is being asserted by 
another agent on the platform. 


¢ Package PROCHOT# Log (bit 3, R/WCO) — Sticky bit that indicates whether package PROCHOT# has been 
asserted by another agent on the platform since the last clearing of this bit or a reset. If bit 3 = 1, package 
PROCHOT# has been externally asserted. Software may clear this bit by writing a zero. 


° Package Critical Temperature Status (bit 4, RO) — Indicates whether the package critical temperature 
detector output signal is currently active. If bit 4 = 1, the package critical temperature detector output signal 
is currently active. 


¢ Package Critical Temperature Log (bit 5, R/ WCO) — Sticky bit that indicates whether the package critical 
temperature detector output signal has been asserted since the last clearing of this bit or reset. If bit 5 = 1, the 
output signal has been asserted. Software may clear this bit by writing a zero. 


¢ Package Thermal Threshold #1 Status (bit 6, RO) — Indicates whether the actual package temperature is 
currently higher than or equal to the value set in Package Thermal Threshold #1. If bit 6 = 0, the actual 
temperature is lower. If bit 6 = 1, the actual temperature is greater than or equal to PTT#1. Quantitative 
information of actual package temperature can be inferred from Package Digital Readout, bits 22:16. 


¢ Package Thermal Threshold #1 Log (bit 7, R/WCO) — Sticky bit that indicates whether the Package 
Thermal Threshold #1 has been reached since the last clearing of this bit or a reset. If bit 7 = 1, the Package 
Threshold #1 has been reached. Software may clear this bit by writing a zero. 


° Package Thermal Threshold #2 Status (bit 8, RO) — Indicates whether actual package temperature is 
currently higher than or equal to the value set in Package Thermal Threshold #2. If bit 8 = 0, the actual 
temperature is lower. If bit 8 = 1, the actual temperature is greater than or equal to PTT#2. Quantitative 
information of actual temperature can be inferred from Package Digital Readout, bits 22:16. 


¢ Package Thermal Threshold #2 Log (bit 9, R/WCO) — Sticky bit that indicates whether the Package 
Thermal Threshold #2 has been reached since the last clearing of this bit or a reset. If bit 9 = 1, the Package 
Thermal Threshold #2 has been reached. Software may clear this bit by writing a zero. 
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¢ Package Power Limitation Status (bit 10, RO) — Indicates package power limit is forcing one ore more 
processors to operate below OS-requested P-state. Note that package power limit violation may be caused by 
processor cores or by devices residing in the uncore. Software can examine IA32_THERM_STATUS to 
determine if the cause originates from a processor core (see Figure 14-31). 


¢ Package Power Notification Log (bit 11, R/WCO) — Sticky bit that indicates any processor in the package 
went below OS-requested P-state or OS-requested clock modulation duty cycle since the last clearing of this or 
RESET. 


° Package Digital Readout (bits 22:16, RO) — Package digital temperature reading in 1 degree Celsius 
relative to the package TCC activation temperature. 


0: Package TCC Activation temperature, 

1: (PTCC Activation - 1) , etc. See the processor’s data sheet for details regarding PTCC activation. 

A lower reading in the Package Digital Readout field (bits 22:16) indicates a higher actual temperature. 
The layout of IA32_PACKAGE_THERM_INTERRUPT MSR is shown in Figure 14-34. 
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Reserved 


Pkg Power Limit Notification ate 
Pkg Threshold #2 Interrupt Enable 
Pkg Threshold #2 Value 
Pkg Threshold #1 Interrupt Enable ————————__ 
Pkg Threshold #1 Value 
Pkg Overheat Interrupt Enable 
Pkg PROCHOT# Interrupt Enable 
Pkg Low Temp. Interrupt Enable 
Pkg High Temp. Interrupt Enable 


Figure 14-34. |IA32_PACKAGE_THERM_INTERRUPT Register 


¢ Package High-Temperature Interrupt Enable (bit 0, R/W) — This bit allows the BIOS to enable the 
generation of an interrupt on the transition from low-temperature to a package high-temperature threshold. 
Bit O = O (default) disables interrupts; bit 0 = 1 enables interrupts. 


¢ Package Low-Temperature Interrupt Enable (bit 1, R/W) — This bit allows the BIOS to enable the 
generation of an interrupt on the transition from high-temperature to a low-temperature (TCC de-activation). 
Bit 1 = 0 (default) disables interrupts; bit 1 = 1 enables interrupts. 


° Package PROCHOT# Interrupt Enable (bit 2, R/W) — This bit allows the BIOS or OS to enable the 
generation of an interrupt when Package PROCHOT# has been asserted by another agent on the platform and 
the Bidirectional Prochot feature is enabled. Bit 2 = 0 disables the interrupt; bit 2 = 1 enables the interrupt. 


¢ Package Critical Temperature Interrupt Enable (bit 4, R/W) — Enables the generation of an interrupt 
when the Package Critical Temperature Detector has detected a critical thermal condition. The recommended 
response to this condition is a system shutdown. Bit 4 = 0 disables the interrupt; bit 4 = 1 enables the 
interrupt. 


° Package Threshold #1 Value (bits 14:8, R/W) — A temperature threshold, encoded relative to the 
Package TCC Activation temperature (using the same format as the Digital Readout). This threshold is 
compared against the Package Digital Readout and is used to generate the Package Thermal Threshold #1 
Status and Log bits as well as the Package Threshold #1 thermal interrupt delivery. 


¢ Package Threshold #1 Interrupt Enable (bit 15, R/W) — Enables the generation of an interrupt when the 
actual temperature crosses the Package Threshold #1 setting in any direction. Bit 15 = 1 enables the interrupt; 
bit 15 = O disables the interrupt. 


¢ Package Threshold #2 Value (bits 22:16, R/W) —A temperature threshold, encoded relative to the PTCC 
Activation temperature (using the same format as the Package Digital Readout). This threshold is compared 
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against the Package Digital Readout and is used to generate the Package Thermal Threshold #2 Status and Log 
bits as well as the Package Threshold #2 thermal interrupt delivery. 


¢ Package Threshold #2 Interrupt Enable (bit 23, R/W) — Enables the generation of an interrupt when the 
actual temperature crosses the Package Threshold #2 setting in any direction. Bit 23 = 1 enables the interrupt; 
bit 23 = O disables the interrupt. 


° Package Power Limit Notification Enable (bit 24, R/W) — Enables the generation of package power 
notification events. 


14.9.1 |Support for Passive and Active cooling 


Passive and active cooling may be controlled by the OS power management agent through ACPI control methods. 
On platforms providing package level thermal management facility described in the previous section, it is recom- 
mended that active cooling (FAN control) should be driven by measuring the package temperature using the 
IA32_PACKAGE_THERM_INTERRUPT MSR. 


Passive cooling (frequency throttling) should be driven by measuring (a) the core and package temperatures, or 
(b) only the package temperature. If measured package temperature led the power management agent to choose 
which core to execute passive cooling, then all cores need to execute passive cooling. Core temperature is 
measured using the IA32_THERMAL_STATUS and IA32_THERMAL_INTERRUPT MSRs. The exact implementation 
details depend on the platform firmware and possible solutions include defining two different thermal zones (one 
for core temperature and passive cooling and the other for package temperature and active cooling). 


14.10 PLATFORM SPECIFIC POWER MANAGEMENT SUPPORT 


This section covers power management interfaces that are not architectural but addresses the power management 
needs of several platform specific components. Specifically, RAPL (Running Average Power Limit) interfaces provide 
mechanisms to enforce power consumption limit. Power limiting usages have specific usages in client and server 
platforms. 


For client platform power limit control and for server platforms used in a data center, the following power and 
thermal related usages are desirable: 


¢ Platform Thermal Management: Robust mechanisms to manage component, platform, and group-level 
thermals, either proactively or reactively (e.g., in response to a platform-level thermal trip point). 


¢ Platform Power Limiting: More deterministic control over the system's power consumption, for example to meet 
battery life targets on rack-level or container-level power consumption goals within a datacenter. 


° Power/Performance Budgeting: Efficient means to control the power consumed (and therefore the sustained 
performance delivered) within and across platforms. 


The server and client usage models are addressed by RAPL interfaces, which expose multiple domains of power 
rationing within each processor socket. Generally, these RAPL domains may be viewed to include hierarchically: 


¢ Package domain is the processor die. 


¢ Memory domain includes the directly-attached DRAM; an additional power plane may constitute a separate 
domain. 


In order to manage the power consumed across multiple sockets via RAPL, individual limits must be programmed 
for each processor complex. Programming specific RAPL domain across multiple sockets is not supported. 


14.10.1  RAPL Interfaces 


RAPL interfaces consist of non-architectural MSRs. Each RAPL domain supports the following set of capabilities, 
some of which are optional as stated below. 


¢ Power limit - MSR interfaces to specify power limit, time window; lock bit, clamp bit etc. 
¢ Energy Status - Power metering interface providing energy consumption information. 
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e  =Perf Status (Optional) - Interface providing information on the performance effects (regression) due to power 
limits. It is defined as a duration metric that measures the power limit effect in the respective domain. The 
meaning of duration is domain specific. 


¢ Power Info (Optional) - Interface providing information on the range of parameters for a given domain, 
minimum power, maximum power etc. 


° Policy (Optional) - 4-bit priority information that is a hint to hardware for dividing budget between sub-domains 
in a parent domain. 


Each of the above capabilities requires specific units in order to describe them. Power is expressed in Watts, Time 
is expressed in Seconds, and Energy is expressed in Joules. Scaling factors are supplied to each unit to make the 
information presented meaningful in a finite number of bits. Units for power, energy, and time are exposed in the 
read-only MSR_RAPL_POWER_UNIT MSR. 
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Reserved 
Time units 
Energy status units 
Power units 


Figure 14-35. MSR_RAPL_POWER_UNIT Register 


MSR_RAPL_POWER_UNIT (Figure 14-35) provides the following information across all RAPL domains: 


° Power Units (bits 3:0): Power related information (in Watts) is based on the multiplier, 1/ 2*PU; where PU is 
an unsigned integer represented by bits 3:0. Default value is 0011b, indicating power unit is in 1/8 Watts 
increment. 


° Energy Status Units (bits 12:8): Energy related information (in Joules) is based on the multiplier, 1/2“ESU; 
where ESU is an unsigned integer represented by bits 12:8. Default value is 10000b, indicating energy status 
unit is in 15.3 micro-Joules increment. 


¢ Time Units (bits 19:16): Time related information (in Seconds) is based on the multiplier, 1/ 2*TU; where TU 
is an unsigned integer represented by bits 19:16. Default value is 1010b, indicating time unit is in 976 micro- 
seconds increment. 


14.10.2 RAPL Domains and Platform Specificity 


The specific RAPL domains available in a platform vary across product segments. Platforms targeting the client 
segment support the following RAPL domain hierarchy: 

° Package 

¢® Two power planes: PPO and PP1 (PP1 may reflect to uncore devices) 

Platforms targeting the server segment support the following RAPL domain hierarchy: 

° Package 

¢ Power plane: PPO 

¢ DRAM 


Each level of the RAPL hierarchy provides a respective set of RAPL interface MSRs. Table 14-8 lists the RAPL MSR 
interfaces available for each RAPL domain. The power limit MSR of each RAPL domain is located at offset 0 relative 
to an MSR base address which is non-architectural (see Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 
64 and IA-32 Architectures Software Developer’s Manual, Volume 4). The energy status MSR of each domain is 
located at offset 1 relative to the MSR base address of respective domain. 
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Table 14-8. RAPL MSR Interfaces and RAPL Domains 


Domain Power Limit Energy Status (Offset Policy Perf Status Power Info 
(Offset 0) 1) (Offset 2) (Offset 3) (Offset 4) 
PKG MSR_PKG_POWER_ | MSR_PKG_ENERGY_STA | RESERVED MSR_PKG_PERF_STATUS MSR_PKG_POWER_| 
LIMIT TUS NFO 
DRAM MSR_DRAM_POWER | MSR_DRAM_ENERGY_S | RESERVED MSR_DRAM_PERF_STATUS | MSR_DRAM_POWER 
_UMIT TATUS _INFO 
PPO MSR_PPO_POWER_ | MSR_PPO_ENERGY_STA | MSR_PPO_POLICY | MSR_PPO_PERF_STATUS RESERVED 
LIMIT TUS 
PP1 MSR_PP1_POWER_ ) MSR_PP1_ENERGY_STA | MSR_PP1_POLICY | RESERVED RESERVED 
LIMIT TUS 


The presence of the optional MSR interfaces (the three right-most columns of Table 14-8) may be model-specific. 
See Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4 for details. 


14.10.3 Package RAPL Domain 


The MSR interfaces defined for the package RAPL domain are: 


¢ MSR_PKG_POWER_LIMIT allows software to set power limits for the package and measurement attributes 
associated with each limit, 


¢ MSR_PKG_ENERGY_STATUS reports measured actual energy usage, 
¢ MSR_PKG_POWER_INFO reports the package power range information for RAPL usage. 


MSR_PKG_PERF_STATUS can report the performance impact of power limiting, but its availability may be model- 
specific. 
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Figure 14-36. MSR_PKG_POWER_LIMIT Register 


MSR_PKG_POWER_LIMIT allows a software agent to define power limitation for the package domain. Power limita- 
tion is defined in terms of average power usage (Watts) over a time window specified in MSR_PKG_POWER_LIMIT. 
Two power limits can be specified, corresponding to time windows of different sizes. Each power limit provides inde- 
pendent clamping control that would permit the processor cores to go below OS-requested state to meet the power 
limits. A lock mechanism allow the software agent to enforce power limit settings. Once the lock bit is set, the 
power limit settings are static and un-modifiable until next RESET. 


The bit fields of MSR_PKG_POWER_LIMIT (Figure 14-36) are: 


° Package Power Limit #1(bits 14:0): Sets the average power usage limit of the package domain corre- 
sponding to time window # 1. The unit of this field is specified by the “Power Units” field of 
MSR_RAPL_POWER_UNIT. 
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¢ Enable Power Limit #1(bit 15): 0 = disabled; 1 = enabled. 


¢ Package Clamping Limitation #1 (bit 16): Allow going below OS-requested P/T state setting during time 
window specified by bits 23:17. 


¢ Time Window for Power Limit #1 (bits 23:17): Indicates the time window for power limit #1 
Time limit = 24Y * (1.0 + Z/4.0) * Time_Unit 


Here “Y” is the unsigned integer value represented. by bits 21:17, “Z” is an unsigned integer represented by 
bits 23:22. “Time_Unit” is specified by the “Time Units” field of MSR_RAPL_POWER_UNIT. 


° Package Power Limit #2(bits 46:32): Sets the average power usage limit of the package domain corre- 
sponding to time window # 2. The unit of this field is specified by the “Power Units” field of 
MSR_RAPL_POWER_UNIT. 


° Enable Power Limit #2(bit 47): 0 = disabled; 1 = enabled. 


¢ Package Clamping Limitation #2 (bit 48): Allow going below OS-requested P/T state setting during time 
window specified by bits 23:17. 


¢ Time Window for Power Limit #2 (bits 55:49): Indicates the time window for power limit #2 
Time limit = 24Y * (1.0 + Z/4.0) * Time_Unit 
Here “Y” is the unsigned integer value represented. by bits 53:49, “Z” is an unsigned integer represented by 
bits 55:54. “Time_Unit” is specified by the “Time Units” field of MSR_RAPL_POWER_UNIT. This field may have 
a hard-coded value in hardware and ignores values written by software. 

°* Lock (bit 63): If set, all write attempts to this MSR are ignored until next RESET. 


MSR_PKG_ENERGY_STATUS is a read-only MSR. It reports the actual energy use for the package domain. This 
MSR is updated every ~1imsec. It has a wraparound time of around 60 secs when power consumption is high, and 
may be longer otherwise. 
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Figure 14-37. MSR_PKG_ENERGY_STATUS MSR 


° Total Energy Consumed (bits 31:0): The unsigned integer value represents the total amount of energy 
consumed since that last time this register is cleared. The unit of this field is specified by the “Energy Status 
Units” field of MSR_RAPL_POWER_UNIT. 


MSR_PKG_POWER_INFO is a read-only MSR. It reports the package power range information for RAPL usage. This 
MSR provides maximum/minimum values (derived from electrical specification), thermal specification power of the 
package domain. It also provides the largest possible time window for software to program the RAPL interface. 
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Figure 14-38. MSR_PKG_POWER_INFO Register 


¢ Thermal Spec Power (bits 14:0): The unsigned integer value is the equivalent of thermal specification power 
of the package domain. The unit of this field is specified by the “Power Units” field of MSR_RAPL_POWER_UNIT. 
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¢ Minimum Power (bits 30:16): The unsigned integer value is the equivalent of minimum power derived from 
electrical spec of the package domain. The unit of this field is specified by the “Power Units” field of 
MSR_RAPL_POWER_UNIT. 


¢ Maximum Power (bits 46:32): The unsigned integer value is the equivalent of maximum power derived from 
the electrical spec of the package domain. The unit of this field is specified by the “Power Units” field of 
MSR_RAPL_POWER_UNIT. 


¢ Maximum Time Window (bits 53:48): The unsigned integer value is the equivalent of largest acceptable 
value to program the time window of MSR_PKG_POWER_LIMIT. The unit of this field is specified by the “Time 
Units” field of MSR_RAPL_POWER_UNIT. 


MSR_PKG_PERF_STATUS is a read-only MSR. It reports the total time for which the package was throttled due to 
the RAPL power limits. Throttling in this context is defined as going below the OS-requested P-state or T-state. It 
has a wrap-around time of many hours. The availability of this MSR is platform specific (see Chapter 2, “Model- 
Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4). 
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Figure 14-39. MSR_PKG_PERF_STATUS MSR 


¢ Accumulated Package Throttled Time (bits 31:0): The unsigned integer value represents the cumulative 
time (since the last time this register is cleared) that the package has throttled. The unit of this field is specified 
by the “Time Units” field of MSR_RAPL_POWER_UNIT. 


14.10.4 PPO/PP1 RAPL Domains 


The MSR interfaces defined for the PPO and PP1 domains are identical in layout. Generally, PPO refers to the 

processor cores. The availability of PP1 RAPL domain interface is platform-specific. For a client platform, the PP1 

domain refers to the power plane of a specific device in the uncore. For server platforms, the PP1 domain is not 

supported, but its PPO domain supports the MSR_PPO_PERF_STATUS interface. 

¢ MSR_PPO_POWER_LIMIT/MSR_PP1_POWER_LIMIT allow software to set power limits for the respective power 
plane domain. 

¢ MSR_PPO_ENERGY_STATUS/MSR_PP1_ENERGY_STATUS report actual energy usage on a power plane. 

¢ MSR_PPO_POLICY/MSR_PP1_POLICY allow software to adjust balance for respective power plane. 


MSR_PPO_PERF_STATUS can report the performance impact of power limiting, but it is not available in client plat- 
forms. 
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Figure 14-40. MSR_PPO_POWER_LIMIT/MSR_PP1_POWER_LIMIT Register 
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MSR_PPO_POWER_LIMIT/MSR_PP1_POWER_LIMIT allow a software agent to define power limitation for the 
respective power plane domain. A lock mechanism in each power plane domain allows the software agent to 
enforce power limit settings independently. Once a lock bit is set, the power limit settings in that power plane are 
static and un-modifiable until next RESET. 


The bit fields of MSR_PPO_POWER_LIMIT/MSR_PP1_POWER_LIMIT (Figure 14-40) are: 

° Power Limit (bits 14:0): Sets the average power usage limit of the respective power plane domain. The unit 
of this field is specified by the “Power Units” field of MSR_RAPL_POWER_UNIT. 

¢ Enable Power Limit (bit 15): 0 = disabled; 1 = enabled. 


¢ Clamping Limitation (bit 16): Allow going below OS-requested P/T state setting during time window specified 
by bits 23:17. 


¢ Time Window for Power Limit (bits 23:17): Indicates the length of time window over which the power limit 
#1 will be used by the processor. The numeric value encoded by bits 23:17 is represented by the product of 
2“Y *F; where F is a single-digit decimal floating-point value between 1.0 and 1.3 with the fraction digit 
represented by bits 23:22, Y is an unsigned integer represented by bits 21:17. The unit of this field is specified 
by the “Time Units” field of MSR_RAPL_POWER_UNIT. 


° Lock (bit 31): If set, all write attempts to the MSR and corresponding policy 
MSR_PPO_POLICY/MSR_PP1_POLICY are ignored until next RESET. 


MSR_PPO_ENERGY_STATUS/MSR_PP1_ENERGY_STATUS are read-only MSRs. They report the actual energy use 
for the respective power plane domains. These MSRs are updated every ~1msec. 
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Figure 14-41. MSR_PPO_ENERGY_STATUS/MSR_PP1_ENERGY_STATUS MSR 


° Total Energy Consumed (bits 31:0): The unsigned integer value represents the total amount of energy 
consumed since the last time this register was cleared. The unit of this field is specified by the “Energy Status 
Units” field of MSR_RAPL_POWER_UNIT. 


MSR_PPO_POLICY/MSR_PP1_POLICY provide balance power policy control for each power plane by providing 
inputs to the power budgeting management algorithm. On platforms that support PPO (IA cores) and PP1 (uncore 
graphic device), the default values give priority to the non-IA power plane. These MSRs enable the PCU to balance 
power consumption between the IA cores and uncore graphic device. 
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Figure 14-42. MSR_PPO_POLICY/MSR_PP1_POLICY Register 


° Priority Level (bits 4:0): Priority level input to the PCU for respective power plane. PPO covers the IA 
processor cores, PP1 covers the uncore graphic device. The value 31 is considered highest priority. 


MSR_PPO_PERF_STATUS is a read-only MSR. It reports the total time for which the PPO domain was throttled due 
to the power limits. This MSR is supported only in server platform. Throttling in this context is defined as going 
below the OS-requested P-state or T-state. 
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Figure 14-43. MSR_PPO_PERF_STATUS MSR 


Accumulated PPO Throttled Time (bits 31:0): The unsigned integer value represents the cumulative time 


(since the last time this register is cleared) that the PPO domain has throttled. The unit of this field is specified 
by the “Time Units” field of MSR_RAPL_POWER_UNIT. 


14.10.5 DRAM RAPL Domain 


The MSR interfaces defined for the DRAM domains are supported only in the server platform. The MSR interfaces 


are: 


MSR_DRAM_POWER_LIMIT allows software to set power limits for the DRAM domain and measurement 
attributes associated with each limit. 


MSR_DRAM_ENERGY_STATUS reports measured actual energy usage. 
MSR_DRAM_POWER_INFO reports the DRAM domain power range information for RAPL usage. 
MSR_DRAM_PERF_STATUS can report the performance impact of power limiting. 
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Figure 14-44. MSR_DRAM_POWER_LIMIT Register 


MSR_DRAM_POWER_LIMIT allows a software agent to define power limitation for the DRAM domain. Power limita- 
tion is defined in terms of average power usage (Watts) over a time window specified in 
MSR_DRAM_POWER_LIMIT. A power limit can be specified along with a time window. A lock mechanism allow the 
software agent to enforce power limit settings. Once the lock bit is set, the power limit settings are static and un- 
modifiable until next RESET. 


The bit fields of MSR_DRAM_POWER_LIMIT (Figure 14-44) are: 


DRAM Power Limit #1(bits 14:0): Sets the average power usage limit of the DRAM domain corresponding to 
time window # 1. The unit of this field is specified by the “Power Units” field of MSR_RAPL_POWER_UNIT. 


Enable Power Limit #1(bit 15): 0 = disabled; 1 = enabled. 


Time Window for Power Limit (bits 23:17): Indicates the length of time window over which the power limit 
will be used by the processor. The numeric value encoded by bits 23:17 is represented by the product of 2“Y 
*F; where F is a single-digit decimal floating-point value between 1.0 and 1.3 with the fraction digit represented 
by bits 23:22, Y is an unsigned integer represented by bits 21:17. The unit of this field is specified by the “Time 
Units” field of MSR_RAPL_POWER_UNIT. 


Lock (bit 31): If set, all write attempts to this MSR are ignored until next RESET. 
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MSR_DRAM_ENERGY_STATUS is a read-only MSR. It reports the actual energy use for the DRAM domain. This MSR 
is updated every ~1msec. 
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Figure 14-45. MSR_DRAM_ENERGY_STATUS MSR 


° Total Energy Consumed (bits 31:0): The unsigned integer value represents the total amount of energy 
consumed since that last time this register is cleared. The unit of this field is specified by the “Energy Status 
Units” field of MSR_RAPL_POWER_UNIT. 


MSR_DRAM_POWER_INFO is a read-only MSR. It reports the DRAM power range information for RAPL usage. This 
MSR provides maximum/minimum values (derived from electrical specification), thermal specification power of the 
DRAM domain. It also provides the largest possible time window for software to program the RAPL interface. 
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Figure 14-46. MSR_DRAM_POWER_INFO Register 


¢ Thermal Spec Power (bits 14:0): The unsigned integer value is the equivalent of thermal specification power 
of the DRAM domain. The unit of this field is specified by the “Power Units” field of MSR_RAPL_POWER_UNIT. 


¢ Minimum Power (bits 30:16): The unsigned integer value is the equivalent of minimum power derived from 
electrical spec of the DRAM domain. The unit of this field is specified by the “Power Units” field of 
MSR_RAPL_POWER_UNIT. 


¢ Maximum Power (bits 46:32): The unsigned integer value is the equivalent of maximum power derived from 
the electrical spec of the DRAM domain. The unit of this field is specified by the “Power Units” field of 
MSR_RAPL_POWER_UNIT. 


¢ Maximum Time Window (bits 53:48): The unsigned integer value is the equivalent of largest acceptable 
value to program the time window of MSR_DRAM_POWER_LIMIT. The unit of this field is specified by the “Time 
Units” field of MSR_RAPL_POWER_UNIT. 


MSR_DRAM_PERF_STATUS is a read-only MSR. It reports the total time for which the package was throttled due to 
the RAPL power limits. Throttling in this context is defined as going below the OS-requested P-state or T-state. It 
has a wrap-around time of many hours. The availability of this MSR is platform specific (see Chapter 2, “Model- 
Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4). 
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Figure 14-47. MSR_DRAM_PERF_STATUS MSR 
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¢ Accumulated Package Throttled Time (bits 31:0): The unsigned integer value represents the cumulative 
time (since the last time this register is cleared) that the DRAM domain has throttled. The unit of this field is 
specified by the “Time Units” field of MSR_RAPL_POWER_UNIT. 
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MACHINE-CHECK ARCHITECTURE 


This chapter describes the machine-check architecture and machine-check exception mechanism found in the 
Pentium 4, Intel Xeon, Intel Atom, and P6 family processors. See Chapter 6, “Interrupt 18—Machine-Check Excep- 
tion (#MC),” for more information on machine-check exceptions. A brief description of the Pentium processor's 
machine check capability is also given. 


Additionally, a signaling mechanism for software to respond to hardware corrected machine check error is covered. 


15.1 MACHINE-CHECK ARCHITECTURE 


The Pentium 4, Intel Xeon, Intel Atom, and P6 family processors implement a machine-check architecture that 
provides a mechanism for detecting and reporting hardware (machine) errors, such as: system bus errors, ECC 
errors, parity errors, cache errors, and TLB errors. It consists of a set of model-specific registers (MSRs) that are 
used to set up machine checking and additional banks of MSRs used for recording errors that are detected. 


The processor signals the detection of an uncorrected machine-check error by generating a machine-check excep- 
tion (#MC), which is an abort class exception. The implementation of the machine-check architecture does not 
ordinarily permit the processor to be restarted reliably after generating a machine-check exception. However, the 
machine-check-exception handler can collect information about the machine-check error from the machine-check 
MSRs. 


Starting with 45 nm Intel 64 processor on which CPUID reports DisplayFamily_DisplayModel as O6H_1AH (see 
CPUID instruction in Chapter 3, “Instruction Set Reference, A-L” in the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 2A), the processor can report information on corrected machine-check errors and 
deliver a programmable interrupt for software to respond to MC errors, referred to as corrected machine-check 
error interrupt (CMCI). See Section 15.5 for detail. 


Intel 64 processors supporting machine-check architecture and CMCI may also support an additional enhance- 
ment, namely, support for software recovery from certain uncorrected recoverable machine check errors. See 
Section 15.6 for detail. 


15.2 COMPATIBILITY WITH PENTIUM PROCESSOR 


The Pentium 4, Intel Xeon, Intel Atom, and P6 family processors support and extend the machine-check exception 
mechanism introduced in the Pentium processor. The Pentium processor reports the following machine-check 
errors: 


¢ data parity errors during read cycles 
* unsuccessful completion of a bus cycle 
The above errors are reported using the PS_MC_TYPE and P5_MC_ADDR MSRs (implementation specific for the 


Pentium processor). Use the RDMSR instruction to read these MSRs. See Chapter 2, “Model-Specific Registers 
(MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4 for the addresses. 


The machine-check error reporting mechanism that Pentium processors use is similar to that used in Pentium 4, 
Intel Xeon, Intel Atom, and P6 family processors. When an error is detected, it is recorded in PS_MC_TYPE and 
P5_MC_ADDR; the processor then generates a machine-check exception (#MC). 


See Section 15.3.3, “Mapping of the Pentium Processor Machine-Check Errors to the Machine-Check Architecture,” 
and Section 15.10.2, “Pentium Processor Machine-Check Exception Handling,” for information on compatibility 
between machine-check code written to run on the Pentium processors and code written to run on P6 family 
processors. 
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15.3. MACHINE-CHECK MSRS 


Machine check MSRs in the Pentium 4, Intel Atom, Intel Xeon, and P6 family processors consist of a set of global 
control and status registers and several error-reporting register banks. See Figure 15-1. 


Global Control MSRs Error-Reporting Bank Registers 
(One Set for Each Hardware Unit) 
63 0 63 0 
IA32_MCG_CAP MSR 1A32_MCi_CTL MSR 
63 0 63 0 
IA32_MCG_STATUS MSR 1A32_MCi_STATUS MSR 
63 0 63 0 
1A32_MCG_CTL MSR 1A32_MCi_ADDR MSR 
63 0 63 0 
IA32_MCG_EXT_CTL MSR 1A32_MCi_MISC MSR 
63 0 
IA32_MCi_CTL2 MSR 


Figure 15-1. Machine-Check MSRs 


Each error-reporting bank is associated with a specific hardware unit (or group of hardware units) in the processor. 
Use RDMSR and WRMSR to read and to write these registers. 


15.3.1 Machine-Check Global Control MSRs 


The machine-check global control MSRs include the IA32_MCG_CAP, IA32_MCG_STATUS, and optionally 
IA32_MCG_CTL and IA32_MCG_EXT_CTL. See Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 4 for the addresses of these registers. 


15.3.1.1 IA32_MCG_CAP MSR 


The IA32_MCG_CAP MSR is a read-only register that provides information about the machine-check architecture of 
the processor. Figure 15-2 shows the layout of the register. 
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MCG_LMCE_P[27] ———— 
MCG_ELOG P[26] 


MCG_EMC _P[25] 
MCG_SER_P[24] 
MCG_EXT_CNT[23:16] 
MCG_TES P[11] 
MCG_CMCI_P[10] 
MCG_EXT_P[9] 
MCG_CTL_P[8] 


Figure 15-2. IA32_MCG_CAP Register 


Where: 


Count field, bits 7:0 — Indicates the number of hardware unit error-reporting banks available in a particular 
processor implementation. 


MCG_CTL_P (control MSR present) flag, bit 8 — Indicates that the processor implements the 
IA32_MCG_CTL MSR when set; this register is absent when clear. 


MCG_EXT_P (extended MSRs present) flag, bit 9 — Indicates that the processor implements the extended 
machine-check state registers found starting at MSR address 180H; these registers are absent when clear. 


MCG_CMCI_P (Corrected MC error counting/signaling extension present) flag, bit 10 — Indicates 
(when set) that extended state and associated MSRs necessary to support the reporting of an interrupt on a 
corrected MC error event and/or count threshold of corrected MC errors, is present. When this bit is set, it does 
not imply this feature is supported across all banks. Software should check the availability of the necessary 
logic on a bank by bank basis when using this signaling capability (i.e. bit 30 settable in individual 
IA32_MCi_CTL2 register). 


MCG_TES_P (threshold-based error status present) flag, bit 11 — Indicates (when set) that bits 56:53 
of the IA32_MCi_STATUS MSR are part of the architectural space. Bits 56:55 are reserved, and bits 54:53 are 
used to report threshold-based error status. Note that when MCG_TES_P is not set, bits 56:53 of the 
IA32_MCi_STATUS MSR are model-specific. 


MCG_EXT_CNT, bits 23:16 — Indicates the number of extended machine-check state registers present. This 
field is meaningful only when the MCG_EXT_P flag is set. 


MCG_SER_P (software error recovery support present) flag, bit 24 — Indicates (when set) that the 
processor supports software error recovery (see Section 15.6), and IA32_MCi_STATUS MSR bits 56:55 are 
used to report the signaling of uncorrected recoverable errors and whether software must take recovery 
actions for uncorrected errors. Note that when MCG_TES_Pis not set, bits 56:53 of the IA32_MCi_STATUS MSR 
are model-specific. If MCG_TES_P is set but MCG_SER_P is not set, bits 56:55 are reserved. 


MCG_EMC_P (Enhanced Machine Check Capability) flag, bit 25 — Indicates (when set) that the 
processor supports enhanced machine check capabilities for firmware first signaling. 


MCG_ELOG_P (extended error logging) flag, bit 26 — Indicates (when set) that the processor allows 
platform firmware to be invoked when an error is detected so that it may provide additional platform specific 
information in an ACPI format “Generic Error Data Entry” that augments the data included in machine check 
bank registers. 


For additional information about extended error logging interface, see 
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/enhanced-mca-logging- 
xeon-paper.pdf. 
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¢ MCG_LMCE_P (local machine check exception) flag, bit 27 — Indicates (when set) that the following 
interfaces are present: 


— an extended state LMCE_S (located in bit 3 of IA32_MCG_STATUS), and 
— the IA32_MCG_EXT_CTL MSR, necessary to support Local Machine Check Exception (LMCE). 


A non-zero MCG_LMCE_P indicates that, when LMCE is enabled as described in Section 15.3.1.5, some machine 
check errors may be delivered to only a single logical processor. 


The effect of writing to the IA32_MCG_CAP MSR is undefined. 


15.3.1.2 IA32_MCG_STATUS MSR 


The IA32_MCG_STATUS MSR describes the current state of the processor after a machine-check exception has 
occurred (see Figure 15-3). 


63 3 


Reserved j 


<U-Mm/}— 
<U-DO 


LMCE_S—Local machine check exception ea 
MCIP—Machine check in progress flag 

EIPV—Error IP valid flag 
RIPV—Restart IP valid flag 


Figure 15-3. IA32_MCG_STATUS Register 


Where: 


° RIPV (restart IP valid) flag, bit O — Indicates (when set) that program execution can be restarted reliably 
at the instruction pointed to by the instruction pointer pushed on the stack when the machine-check exception 
is generated. When clear, the program cannot be reliably restarted at the pushed instruction pointer. 


¢ EIPV (error IP valid) flag, bit 1 — Indicates (when set) that the instruction pointed to by the instruction 
pointer pushed onto the stack when the machine-check exception is generated is directly associated with the 
error. When this flag is cleared, the instruction pointed to may not be associated with the error. 


¢ MCIP (machine check in progress) flag, bit 2 — Indicates (when set) that a machine-check exception was 
generated. Software can set or clear this flag. The occurrence of a second Machine-Check Event while MCIP is 
set will cause the processor to enter a shutdown state. For information on processor behavior in the shutdown 
state, please refer to the description in Chapter 6, “Interrupt and Exception Handling”: “Interrupt 8—Double 
Fault Exception (#DF)”. 


° LMCE_S (local machine check exception signaled), bit 3 — Indicates (when set) that a local machine- 
check exception was generated. This indicates that the current machine-check event was delivered to only this 
logical processor. 


Bits 63:04 in IA32_MCG_STATUS are reserved. An attempt to write to IA32_MCG_STATUS with any value other 
than O would result in #GP. 


15.3.1.3. IA32_MCG_CTL MSR 
The IA32_MCG_CTL MSR is present if the capability flag MCG_CTL_P is set in the IA32_MCG_CAP MSR. 


IA32_MCG_CTL controls the reporting of machine-check exceptions. If present, writing 1s to this register enables 
machine-check features and writing all Os disables machine-check features. All other values are undefined and/or 
implementation specific. 
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15.3.1.4 IA32_MCG_EXT_CTL MSR 
The IA32_MCG_EXT_CTL MSR is present if the capability flag MCG_LMCE_P is set in the IA32_MCG_CAP MSR. 


IA32_MCG_EXT_CTL.LMCE_EN (bit 0) allows the processor to signal some MCEs to only a single logical processor 
in the system. 


If MCG_LMCE_P is not set in IA32_MCG_CAP, or platform software has not enabled LMCE by setting 
IA32_FEATURE_CONTROL.LMCE_ENABLED (bit 20), any attempt to write or read IA32_MCG_EXT_CTL will result in 
#GP. 


The IA32_MCG_EXT_CTL MSR is cleared on RESET. 
Figure 15-4 shows the layout of the IA32_MCG_EXT_CTL register 


63 1.0 


Reserved 


LMCE_EN - system software control to enable/disable LMCE | 


Figure 15-4. IA32_MCG_EXT_CTL Register 


where 


¢ LMCE_EN (local machine check exception enable) flag, bit 0 - System software sets this to allow 
hardware to signal some MCEs to only a single logical processor. System software can set LMCE_EN only if the 
platform software has configured IA32_FEATURE_CONTROL as described in Section 15.3.1.5. 


15.3.1.5 Enabling Local Machine Check 


The intended usage of LMCE requires proper configuration by both platform software and system software. Plat- 
form software can turn LMCE on by setting bit 20 (LMCE_ENABLED) in IA32_FEATURE_CONTROL MSR (MSR 
address 3AH). 


System software must ensure that both IA32_FEATURE_CONTROL.Lock (bit 0)and 
IA32_FEATURE_CONTROL.LMCE_ENABLED (bit 20) are set before attempting to set 
IA32_MCG_EXT_CTL.LMCE_EN (bit 0). When system software has enabled LMCE, then hardware will determine if 
a particular error can be delivered only to a single logical processor. Software should make no assumptions about 
the type of error that hardware can choose to deliver as LMCE. The severity and override rules stay the same as 
described in Table 15-8 to determine the recovery actions. 


15.3.2 Error-Reporting Register Banks 


Each error-reporting register bank can contain the IA32_MCi_CTL, IA32_MCi_STATUS, IA32_MCi_ADDR, and 
IA32_MCi_MISC MSRs. The number of reporting banks is indicated by bits [7:0] of IA32_MCG_CAP MSR (address 
0179H). The first error-reporting register (IA32_MCO_CTL) always starts at address 400H. 


See Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4 for addresses of the error-reporting registers in the Pentium 4, Intel Atom, and Intel Xeon 
processors; and for addresses of the error-reporting registers P6 family processors. 


15.3.2.1  IA32_MCi_CTL MSRs 


The IA32_MCi_CTL MSR controls signaling of #MC for errors produced by a particular hardware unit (or group of 
hardware units). Each of the 64 flags (EEj) represents a potential error. Setting an EEj flag enables signaling #MC 
of the associated error and clearing it disables signaling of the error. Error logging happens regardless of the setting 
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of these bits. The processor drops writes to bits that are not implemented. Figure 15-5 shows the bit fields of 
IA32_MCi_CTL. 


63 62 61 3210 

E 
6\6\6 oe e@ e@ ololo 
3/2)1 2/1/|0 


EEj—Error reporting enable flag 
(where j is 00 through 63) 


Figure 15-5. IA32_MCi_CTL Register 


NOTE 


For P6 family processors, processors based on Intel Core microarchitecture (excluding those on 
which on which CPUID reports DisplayFamily_DisplayModel as O6H_1AH and onward): the 
operating system or executive software must not modify the contents of the IA32_MCO_CTL MSR. 
This MSR is internally aliased to the EBL_CR_POWERON MSR and controls platform-specific error 
handling features. System specific firmware (the BIOS) is responsible for the appropriate initial- 
ization of the IA32_MCO_CTL MSR. P6 family processors only allow the writing of all 1s or all Os to 
the IA32_MCi_CTL MSR. 


15.3.2.2 IA32_MCi_STATUS MSRS 


Each IA32_MCi_STATUS MSR contains information related to a machine-check error if its VAL (valid) flag is set (see 
Figure 15-6). Software is responsible for clearing IA32_MCi_STATUS MSRs by explicitly writing Os to them; writing 
1s to them causes a general-protection exception. 


NOTE 


Figure 15-6 depicts the IA32_MCi_STATUS MSR when IA32_MCG_CAP[24] = 1, 
IA32_MCG_CAP[11] = 1 and IA32_MCG_CAP[10] = 1. When IA32_MCG_CAP[24] = 0 and 
IA32_MCG_CAP[11] = 1, bits 56:55 is reserved and bits 54:53 for threshold-based error reporting. 
When IA32_MCG_CAP[11] = 0, bits 56:53 are part of the “Other Information” field. The use of bits 
54:53 for threshold-based error reporting began with Intel Core Duo processors, and is currently 
used for cache memory. See Section 15.4, “Enhanced Cache Error reporting,” for more information. 
When IA32_MCG_CAP[10] = 0, bits 52:38 are part of the “Other Information” field. The use of bits 
52:38 for corrected MC error count is introduced with Intel 64 processor on which CPUID reports 
DisplayFamily_DisplayModel as O6H_1AH. 


Where: 


° MCA (machine-check architecture) error code field, bits 15:0 — Specifies the machine-check archi- 
tecture-defined error code for the machine-check error condition detected. The machine-check architecture- 
defined error codes are guaranteed to be the same for all IA-32 processors that implement the machine-check 
architecture. See Section 15.9, “Interpreting the MCA Error Codes,” and Chapter 16, “Interpreting Machine- 
Check Error Codes”, for information on machine-check error codes. 


* Model-specific error code field, bits 31:16 — Specifies the model-specific error code that uniquely 
identifies the machine-check error condition detected. The model-specific error codes may differ among IA-32 
processors for the same machine-check error condition. See Chapter 16, “Interpreting Machine-Check Error 
Codes”for information on model-specific error codes. 


® Reserved, Error Status, and Other Information fields, bits 56:32 — 


¢ If IA32_MCG_CAP.MCG_EMC_P[bit 25] is 0, bits 37:32 contain “Other Information” that is implemen- 
tation-specific and is not part of the machine-check architecture. 


¢ If IA832_MCG_CAP.MCG_EMC_P is 1, “Other Information” is in bits 36:32. If bit 37 is 0, system firmware 
has not changed the contents of IA32_MCi_STATUS. If bit 37 is 1, system firmware may have edited the 
contents of IA32_MCi_STATUS. 
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63 62 61 60 59 58 57 56 55 54 53 52 3837 36 32 31 1615 0 
7 a : e s : Corrected Error Other | MSCOD Model MCA Error Code 
Lie c Count Info | Specific Error Code 

R 


|__ Firmware updated error status indicator (37)* 
Threshold-based error status (54:53)** 

AR — Recovery action required for UCR error (55)*** 

S — Signaling an uncorrected recoverable (UCR) error (56)*** 

PCC — Processor context corrupted (57) 

ADDRV — MCi_ADDR register valid (58) 

MISCV — MCi_MISC register valid (59) 

EN — Error reporting enabled (60) 

UC — Uncorrected error (61) 

OVER — Error overflow (62) 

VAL — MCi_STATUS register valid (63) 


* When IA32_MCG_CAP[25] (MCG_EMC_P) is set, bit 37 is not part of “Other Information’. 


** When IA32_MCG_CAP[11] (MCG_TES_P) is not set, these bits are model-specific 
(part of “Other Information’). 


** When IA32_MCG_CAP[11] or [A32_MCG_CAP[24] are not set, these bits are reserved, or 
model-specific (part of “Other Information’). 


Figure 15-6. IA32_MCi_STATUS Register 


If IA32_MCG_CAP.MCG_CMCI_P[bit 10] is 0, bits 52:38 also contain “Other Information” (in the same 
sense as bits 37:32). 


If IA32_MCG_CAP[10] is 1, bits 52:38 are architectural (not model-specific). In this case, bits 52:38 
reports the value of a 15 bit counter that increments each time a corrected error is observed by the MCA 
recording bank. This count value will continue to increment until cleared by software. The most 
significant bit, 52, is a sticky count overflow bit. 


If IA32_MCG_CAP[11] is 0, bits 56:53 also contain “Other Information” (in the same sense). 


If IA32_MCG_CAP[11] is 1, bits 56:53 are architectural (not model-specific). In this case, bits 56:53 
have the following functionality: 


If IA32_MCG_CAP[24] is 0, bits 56:55 are reserved. 
If IA32_MCG_CAP[24] is 1, bits 56:55 are defined as follows: 


S (Signaling) flag, bit 56 - Signals the reporting of UCR errors in this MC bank. See Section 15.6.2 
for additional detail. 


AR (Action Required) flag, bit 55 - Indicates (when set) that MCA error code specific recovery 
action must be performed by system software at the time this error was signaled. See Section 
15.6.2 for additional detail. 


If the UC bit (Figure 15-6) is 1, bits 54:53 are undefined. 


If the UC bit (Figure 15-6) is 0, bits 54:53 indicate the status of the hardware structure that 
reported the threshold-based error. See Table 15-1. 


Table 15-1. Bits 54:53 in IA32_MCi_STATUS MSRs when IA32_MCG_CAP[11] = 1 and UC = 0 


Bits 54:53 | Meaning 
00 No tracking - No hardware status tracking is provided for the structure reporting this event. 
01 Green - Status tracking is provided for the structure posting the event; the current status is green (below threshold). 
For more information, see Section 15.4, “Enhanced Cache Error reporting”. 
10 Yellow - Status tracking is provided for the structure posting the event; the current status is yellow (above threshold). 
For more information, see Section 15.4, “Enhanced Cache Error reporting”. 
11 Reserved 
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¢ PCC (processor context corrupt) flag, bit 57 — Indicates (when set) that the state of the processor might 
have been corrupted by the error condition detected and that reliable restarting of the processor may not be 
possible. When clear, this flag indicates that the error did not affect the processor’s state, and software may be 
able to restart. When system software supports recovery, consult Section 15.10.4, “Machine-Check Software 
Handler Guidelines for Error Recovery” for additional rules that apply. 


¢ ADDRV (IA32_MCi_ADDR register valid) flag, bit 58 — Indicates (when set) that the IA32_MCi_ADDR 
register contains the address where the error occurred (see Section 15.3.2.3, “IA32_MCi_ADDR MSRs”). When 
clear, this flag indicates that the IA32_MCi_ADDR register is either not implemented or does not contain the 
address where the error occurred. Do not read these registers if they are not implemented in the processor. 


° MISCV (IA32_MCi_MISC register valid) flag, bit 59 — Indicates (when set) that the IA32_MCi_MISC 
register contains additional information regarding the error. When clear, this flag indicates that the 
IA32_MCi_MISC register is either not implemented or does not contain additional information regarding the 
error. Do not read these registers if they are not implemented in the processor. 


¢ EN (error enabled) flag, bit 60 — Indicates (when set) that the error was enabled by the associated EEj bit 
of the IA32_MCi_CTL register. 


¢ UC (error uncorrected) flag, bit 61 — Indicates (when set) that the processor did not or was not able to 
correct the error condition. When clear, this flag indicates that the processor was able to correct the error 
condition. 


¢ OVER (machine check overflow) flag, bit 62 — Indicates (when set) that a machine-check error occurred 
while the results of a previous error were still in the error-reporting register bank (that is, the VAL bit was 
already set in the IA32_MCi_STATUS register). The processor sets the OVER flag and software is responsible for 
clearing it. In general, enabled errors are written over disabled errors, and uncorrected errors are written over 
corrected errors. Uncorrected errors are not written over previous valid uncorrected errors. When 
MCG_CMCI_P is set, corrected errors may not set the OVER flag. Software can rely on corrected error count in 
IA32_MCi_Status[52:38] to determine if any additional corrected errors may have occurred. For more infor- 
mation, see Section 15.3.2.2.1, “Overwrite Rules for Machine Check Overflow”. 


° VAL (IA32_MCi_STATUS register valid) flag, bit 63 — Indicates (when set) that the information within the 
IA32_MCi_STATUS register is valid. When this flag is set, the processor follows the rules given for the OVER flag 
in the IA32_MCi_STATUS register when overwriting previously valid entries. The processor sets the VAL flag 
and software is responsible for clearing it. 


15.3.2.2.1 Overwrite Rules for Machine Check Overflow 


Table 15-2 shows the overwrite rules for how to treat a second event if the cache has already posted an event to 
the MC bank - that is, what to do if the valid bit for an MC bank already is set to 1. When more than one structure 
posts events in a given bank, these rules specify whether a new event will overwrite a previous posting or not. 
These rules define a priority for uncorrected (highest priority), yellow, and green/unmonitored (lowest priority) 
status. 


In Table 15-2, the values in the two left-most columns are IA32_MCi_STATUS[54:53]. 


Table 15-2. Overwrite Rules for Enabled Errors 
First Event Second Event UC bit Color MCA Info 
00/green 00/green 0 00/green either 
00/green yellow 0 yellow second error 
yellow 00/green 0 yellow first error 
yellow yellow 0 yellow either 
00/green/yellow UC 1 undefined second 
UC 00/green/yellow 1 undefined first 


If a second event overwrites a previously posted event, the information (as guarded by individual valid bits) in the 
MCi bank is entirely from the second event. Similarly, if a first event is retained, all of the information previously 
posted for that event is retained. In general, when the logged error or the recent error is a corrected error, the 
OVER bit (MCi_Status[62]) may be set to indicate an overflow. When MCG_CMCI_P is set in IA32_MCG_CAP, 
system software should consult IA32_MCi_STATUS[52:38] to determine if additional corrected errors may have 
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occurred. Software may re-read IA32_MCi_STATUS, IA32_MCi_ADDR and IA32_MCi_MISC appropriately to ensure 
data collected represent the last error logged. 


After software polls a posting and clears the register, the valid bit is no longer set and therefore the meaning of the 
rest of the bits, including the yellow/green/O0 status field in bits 54:53, is undefined. The yellow/green indication 
will only be posted for events associated with monitored structures - otherwise the unmonitored (00) code will be 
posted in IA32_MCi_STATUS[54:53]. 


15.3.2.3. IA32_MCi_ADDR MSRs 


The IA32_MCi_ADDR MSR contains the address of the code or data memory location that produced the machine- 
check error if the ADDRV flag in the IA32_MCi_STATUS register is set (see Section 15-7, “IA32_MCi_ADDR MSR”). 
The IA32_MCi_ADDR register is either not implemented or contains no address if the ADDRV flag in the 
IA32_MCi_STATUS register is clear. When not implemented in the processor, all reads and writes to this MSR will 
cause a general protection exception. 


The address returned is an offset into a segment, linear address, or physical address. This depends on the error 
encountered. When these registers are implemented, these registers can be cleared by explicitly writing Os to 
these registers. Writing 1s to these registers will cause a general-protection exception. See Figure 15-7. 


Processor Without Support For Intel 64 Architecture 
63 36 35 0 


Reserved Address 


Processor With Support for Intel 64 Architecture 
63 0 


Address" 


* Useful bits in this field depend on the address methodology in use when the 
the register state is saved. 


Figure 15-7. [A32_MCi_ADDR MSR 


15.3.2.4 IA32_MCi_MISC MSRs 


The IA32_MCi_MISC MSR contains additional information describing the machine-check error if the MISCV flag in 
the IA32_MCi_STATUS register is set. The IA32_MCi_MISC_MSR is either not implemented or does not contain 
additional information if the MISCV flag in the IA32_MCi_STATUS register is clear. 


When not implemented in the processor, all reads and writes to this MSR will cause a general protection exception. 
When implemented in a processor, these registers can be cleared by explicitly writing all Os to them; writing 1s to 
them causes a general-protection exception to be generated. This register is not implemented in any of the error- 
reporting register banks for the P6 or Intel Atom family processors. 


If both MISCV and IA32_MCG_CAP[24] are set, the IA32_MCi_MISC_MSR is defined according to Figure 15-8 to 
support software recovery of uncorrected errors (see Section 15.6). 
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63 98 65 0 


Model Specific Information 


Address Mode 
Recoverable Address LSB 


Figure 15-8. UCR Support in IA32_MCi_MISC Register 


° Recoverable Address LSB (bits 5:0): The lowest valid recoverable address bit. Indicates the position of the least 
significant bit (LSB) of the recoverable error address. For example, if the processor logs bits [43:9] of the 
address, the LSB sub-field in IA32_MCi_MISC is 01001b (9 decimal). For this example, bits [8:0] of the 
recoverable error address in IA32_MCi_ADDR should be ignored. 

e Address Mode (bits 8:6): Address mode for the address logged in IA32_MCi_ADDR. The supported address 
modes are given in Table 15-3. 


Table 15-3. Address Mode in |A32_MCi_MISC[8:6] 


IA32_MCi_MISC[8:6] Encoding Definition 
000 Segment Offset 
001 Linear Address 
010 Physical Address 
011 Memory Address 
100 to 110 Reserved 
111 Generic 


¢ Model Specific Information (bits 63:9): Not architecturally defined. 


15.3.2.4.2 IOMCA 


Logging and Signaling of errors from PCI Express domain is governed by PCI Express Advanced Error Reporting 
(AER) architecture. PCI Express architecture divides errors in two categories: Uncorrectable errors and Correctable 
errors. Uncorrectable errors can further be classified as Fatal or Non-Fatal. Uncorrected IO errors are signaled to 
the system software either as AER Message Signaled Interrupt (MSI) or via platform specific mechanisms such as 
NMI. Generally, the signaling mechanism is controlled by BIOS and/or platform firmware. Certain processors 
support an error handling mode, called IOMCA mode, where Uncorrected PCI Express errors are signaled in the 
form of machine check exception and logged in machine check banks. 


When a processor is in this mode, Uncorrected PCI Express errors are logged in the MCACOD field of the 
IA32_MCi_STATUS register as Generic I/O error. The corresponding MCA error code is defined in Table 15-8. 
IA32_MCi_Status [15:0] Simple Error Code Encoding. Machine check logging complements and does not replace 
AER logging that occurs inside the PCI Express hierarchy. The PCI Express Root Complex and Endpoints continue to 
log the error in accordance with PCI Express AER mechanism. In IOMCA mode, MCi_MISC register in the bank that 
logged IOMCA can optionally contain information that link the Machine Check logs with the AER logs or proprietary 
logs. In such a scenario, the machine check handler can utilize the contents of MCi_MISC to locate the next level of 
error logs corresponding to the same error. Specifically, if MCi_Status.MISCV is 1 and MCACOD is OxOEOB, 
MCi_MISC contains the PCI Express address of the Root Complex device containing the AER Logs. Software can 
consult the header type and class code registers in the Root Complex device's PCIe Configuration space to deter- 
mine what type of device it is. This Root Complex device can either be a PCI Express Root Port, PCI Express Root 
Complex Event Collector or a proprietary device. 
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Errors that originate from PCI Express or Legacy Endpoints are logged in the corresponding Root Port in addition to 
the generating device. If MISCV=1 and MCi_MISC contains the address of the Root Port or a Root Complex Event 
collector, software can parse the AER logs to learn more about the error. 


If MISCV=1 and MCi_MISC points to a device that is neither a Root Complex Event Collector not a Root Port, soft- 
ware must consult the Vendor ID/Device ID and use device specific knowledge to locate and interpret the error log 
registers. In some cases, the Root Complex device configuration space may not be accessible to the software and 
both the Vendor and Device ID read as OxFFFF. 


¢ The format of MCi_MISC for IOMCA errors is shown in Table 15-4. 


Table 15-4. Address Mode in IA32_MCi_MISC[8:6] 


63:40 39:32 31:16 15:9 8:6 5:0 
RSVD PCI Express Segment PCI Express RSVD ADDR MODE" RECOV ADDR LSB! 
number Requestor ID 
NOTES: 


1. Not Applicable if ADDRV=O. 


Refer to PCI Express Specification 3.0 for definition of PCI Express Requestor ID and AER architecture. Refer to PCI 
Firmware Specification 3.0 for an explanation of PCI Ex-press Segment number and how software can access 
configuration space of a PCI Ex-press device given the segment number and Requestor ID. 


15.3.2.5 IA32_MCi_CTL2 MSRs 


The IA32_MCi_CTL2 MSR provides the programming interface to use corrected MC error signaling capability that is 
indicated by IA32_MCG_CAP[10] = 1. Software must check for the presence of IA32_MCi_CTL2 on a per-bank 
basis. 

When IA32_MCG_CAP[10] = 1, the IA32_MCi_CTL2 MSR for each bank exists, i.e. reads and writes to these MSR 
are supported. However, signaling interface for corrected MC errors may not be supported in all banks. 


The layout of IA32_MCi_CTL2 is shown in Figure 15-9: 


63 3130 29 15 14 0 


Reserved Reserved 


CMCI_EN—Enable/disable CMCI_ —~ 
Corrected error count threshold 


Figure 15-9. IA32_MCi_CTL2 Register 


¢ Corrected error count threshold, bits 14:0 — Software must initialize this field. The value is compared with 
the corrected error count field in IA32_MCi_STATUS, bits 38 through 52. An overflow event is signaled to the 
CMCI LVT entry (see Table 10-1) in the APIC when the count value equals the threshold value. The new LVT 
entry in the APIC is at O2FOH offset from the APIC_BASE. If CMCI interface is not supported for a particular 
bank (but IA32_MCG_CAP[10] = 1), this field will always read 0. 

¢ CMCI_EN (Corrected error interrupt enable/disable/indicator), bits 30 — Software sets this bit to 
enable the generation of corrected machine-check error interrupt (CMCI). If CMCI interface is not supported for 
a particular bank (but IA32_MCG_CAP[10] = 1), this bit is writeable but will always return O for that bank. This 
bit also indicates CMCI is supported or not supported in the corresponding bank. See Section 15.5 for details of 
software detection of CMCI facility. 
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Some microarchitectural sub-systems that are the source of corrected MC errors may be shared by more than one 
logical processors. Consequently, the facilities for reporting MC errors and controlling mechanisms may be shared 
by more than one logical processors. For example, the IA32_MCi_CTL2 MSR is shared between logical processors 
sharing a processor core. Software is responsible to program IA32_MCi_CTL2 MSR in a consistent manner with 
CMCI delivery and usage. 


After processor reset, IA32_MCi_CTL2 MSRs are zero’ed. 


15.3.2.6 IA32_MCG Extended Machine Check State MSRs 
The Pentium 4 and Intel Xeon processors implement a variable number of extended machine-check state MSRs. 
The MCG_EXT_P flag in the IA32_MCG_CAP MSR indicates the presence of these extended registers, and the 


MCG_EXT_CNT field indicates the number of these registers actually implemented. See Section 15.3.1.1, 
“TA32_MCG_CAP MSR.” Also see Table 15-5. 


Table 15-5. Extended Machine Check State MSRs 
in Processors Without Support for Intel 64 Architecture 


MSR Address Description 
IA32_MCG_EAX 180H Contains state of the EAX register at the time of the machine-check error. 
IA32_MCG_EBX 181H Contains state of the EBX register at the time of the machine-check error. 
1A32_MCG_ECX 182H Contains state of the ECX register at the time of the machine-check error. 
IA32_MCG_EDX 183H Contains state of the EDX register at the time of the machine-check error. 
1A32_MCG_ESI 184H Contains state of the ESI register at the time of the machine-check error. 
IA32_MCG_EDI 185H Contains state of the EDI register at the time of the machine-check error. 
IA32_MCG_EBP 186H Contains state of the EBP register at the time of the machine-check error. 
IA32_MCG_ESP 187H Contains state of the ESP register at the time of the machine-check error. 
IA32_MCG_EFLAGS 188H Contains state of the EFLAGS register at the time of the machine-check error. 
IA32_MCG_EIP 189H Contains state of the EIP register at the time of the machine-check error. 
IA32_MCG_MISC 18AH When set, indicates that a page assist or page fault occurred during DS normal 
operation. 


In processors with support for Intel 64 architecture, 64-bit machine check state MSRs are aliased to the legacy 
MSRs. In addition, there may be registers beyond IA32_MCG_MISC. These may include up to five reserved MSRs 
(IA32_MCG_RESERVED[1:5]) and save-state MSRs for registers introduced in 64-bit mode. See Table 15-6. 


Table 15-6. Extended Machine Check State MSRs 
In Processors With Support For Intel 64 Architecture 


MSR Address Description 

IA32_MCG_RAX 180H Contains state of the RAX register at the time of the machine-check error. 
IA32_MCG_RBX 181H Contains state of the RBX register at the time of the machine-check error. 
IA32_MCG_RCX 182H Contains state of the RCX register at the time of the machine-check error. 
IA32_MCG_RDX 183H Contains state of the RDX register at the time of the machine-check error. 
IA32_MCG_RSI 184H Contains state of the RSI register at the time of the machine-check error. 
IA32_MCG_RDI 185H Contains state of the RDI register at the time of the machine-check error. 
IA32_MCG_RBP 186H Contains state of the RBP register at the time of the machine-check error. 
IA32_MCG_RSP 187H Contains state of the RSP register at the time of the machine-check error. 
IA32_MCG_RFLAGS 188H Contains state of the RFLAGS register at the time of the machine-check error. 
IA32_MCG_RIP 189H Contains state of the RIP register at the time of the machine-check error. 
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Table 15-6. Extended Machine Check State MSRs 
In Processors With Support For Intel 64 Architecture (Contd.) 


MSR Address Description 

IA32_MCG_MISC 18AH When set, indicates that a page assist or page fault occurred during DS normal 
operation. 

IA32_MCG_ 18BH- These registers, if present, are reserved. 

RSERVED[1:5] 18FH 

1A32_MCG_R8 190H Contains state of the R8 register at the time of the machine-check error. 

1A32_MCG_R9 191H Contains state of the R9 register at the time of the machine-check error. 

IA32_MCG_R10 192H Contains state of the R10 register at the time of the machine-check error. 

IA32_MCG_R11 193H Contains state of the R11 register at the time of the machine-check error. 

IA32_MCG_R12 194H Contains state of the R12 register at the time of the machine-check error. 

IA32_MCG_R13 195H Contains state of the R13 register at the time of the machine-check error. 

IA32_MCG_R14 196H Contains state of the R14 register at the time of the machine-check error. 

IA32_MCG_R15 197H Contains state of the R15 register at the time of the machine-check error. 


When a machine-check error is detected on a Pentium 4 or Intel Xeon processor, the processor saves the state of 
the general-purpose registers, the R/EFLAGS register, and the R/EIP in these extended machine-check state MSRs. 
This information can be used by a debugger to analyze the error. 


These registers are read/write to zero registers. This means software can read them; but if software writes to 
them, only all zeros is allowed. If software attempts to write a non-zero value into one of these registers, a general- 
protection (#GP) exception is generated. These registers are cleared on a hardware reset (power-up or RESET), 
but maintain their contents following a soft reset (INIT reset). 


15.3.3. Mapping of the Pentium Processor Machine-Check Errors 
to the Machine-Check Architecture 


The Pentium processor reports machine-check errors using two registers: PS_MC_TYPE and P5_MC_ADDR. The 
Pentium 4, Intel Xeon, Intel Atom, and P6 family processors map these registers to the IA32_MCi_STATUS and 

IA32_MCi_ADDR in the error-reporting register bank. This bank reports on the same type of external bus errors 
reported in PS_MC_TYPE and P5_MC_ADDR. 


The information in these registers can then be accessed in two ways: 


° By reading the IA32_MCi_STATUS and IA32_MCi_ADDR registers as part of a general machine-check exception 
handler written for Pentium 4, Intel Atom and P6 family processors. 


° By reading the P5_MC_TYPE and P5_MC_ADDR registers using the RDMSR instruction. 


The second capability permits a machine-check exception handler written to run on a Pentium processor to be run 
on a Pentium 4, Intel Xeon, Intel Atom, or P6 family processor. There is a limitation in that information returned by 
the Pentium 4, Intel Xeon, Intel Atom, and P6 family processors is encoded differently than information returned 
by the Pentium processor. To run a Pentium processor machine-check exception handler on a Pentium 4, Intel 
Xeon, Intel Atom, or P6 family processor; the handler must be written to interpret PS_MC_TYPE encodings 
correctly. 


15.4 | ENHANCED CACHE ERROR REPORTING 


Starting with Intel Core Duo processors, cache error reporting was enhanced. In earlier Intel processors, cache 
status was based on the number of correction events that occurred in a cache. In the new paradigm, called 
“threshold-based error status”, cache status is based on the number of lines (ECC blocks) in a cache that incur 
repeated corrections. The threshold is chosen by Intel, based on various factors. If a processor supports threshold- 
based error status, it sets IA32_MCG_CAP[11] (MCG_TES_P) to 1; if not, to 0. 
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A processor that supports enhanced cache error reporting contains hardware that tracks the operating status of 
certain caches and provides an indicator of their “health”. The hardware reports a “green” status when the number 
of lines that incur repeated corrections is at or below a pre-defined threshold, and a “yellow” status when the 
number of affected lines exceeds the threshold. Yellow status means that the cache reporting the event is operating 
correctly, but you should schedule the system for servicing within a few weeks. 


Intel recommends that you rely on this mechanism for structures supported by threshold-base error reporting. 


The CPU/system/platform response to a yellow event should be less severe than its response to an uncorrected 
error. An uncorrected error means that a serious error has actually occurred, whereas the yellow condition is a 
warning that the number of affected lines has exceeded the threshold but is not, in itself, a serious event: the error 
was corrected and system state was not compromised. 


The green/yellow status indicator is not a foolproof early warning for an uncorrected error resulting from the failure 
of two bits in the same ECC block. Such a failure can occur and cause an uncorrected error before the yellow 
threshold is reached. However, the chance of an uncorrected error increases as the number of affected lines 
increases. 


15.5 CORRECTED MACHINE CHECK ERROR INTERRUPT 


Corrected machine-check error interrupt (CMCI) is an architectural enhancement to the machine-check architec- 
ture. It provides capabilities beyond those of threshold-based error reporting (Section 15.4). With threshold-based 
error reporting, software is limited to use periodic polling to query the status of hardware corrected MC errors. 
CMCI provides a signaling mechanism to deliver a local interrupt based on threshold values that software can 
program using the IA32_MCi_CTL2 MSRs. 


CMCI is disabled by default. System software is required to enable CMCI for each IA32_MCi bank that support the 
reporting of hardware corrected errors if IA32_MCG_CAP[10] = 1. 


System software use IA32_MCi_CTL2 MSR to enable/disable the CMCI capability for each bank and program 
threshold values into IA32_MCi_CTL2 MSR. CMCI is not affected by the CR4.MCE bit, and it is not affected by the 
IA32_MCi_CTL MSRs. 


To detect the existence of thresholding for a given bank, software writes only bits 14:0 with the threshold value. If 
the bits persist, then thresholding is available (and CMCI is available). If the bits are all O's, then no thresholding 
exists. To detect that CMCI signaling exists, software writes a 1 to bit 30 of the MCi_CTL2 register. Upon subsequent 
read, if bit 30 = 0, no CMCI is available for this bank and no corrected or UCNA errors will be reported on this bank. 
If bit 30 = 1, then CMCI is available and enabled. 


15.5.1 | CMCI Local APIC Interface 
The operation of CMCI is depicted in Figure 15-10. 


Software write 1 to enable 


63 31 30 29 14 0 
MCi_CTL2 Error threshold 


Count overflow threshold -> CMCI LVT in local APIC 
a 
APIC_BASE + 2F0H 


53 52 38 37 0 
T 
MCi_STATUS | Error count | 
| 


Figure 15-10. CMCI Behavior 
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CMCI interrupt delivery is configured by writing to the LVT CMCI register entry in the local APIC register space at 
default address of APIC_BASE + 2FOH. A CMCI interrupt can be delivered to more than one logical processors if 
multiple logical processors are affected by the associated MC errors. For example, if a corrected bit error in a cache 
shared by two logical processors caused a CMCI, the interrupt will be delivered to both logical processors sharing 
that microarchitectural sub-system. Similarly, package level errors may cause CMCI to be delivered to all logical 
processors within the package. However, system level errors will not be handled by CMCI. 


See Section 10.5.1, “Local Vector Table” for details regarding the LVT CMCI register. 


15.5.2 System Software Recommendation for Managing CMCI and Machine Check Resources 


System software must enable and manage CMCI, set up interrupt handlers to service CMCI interrupts delivered to 
affected logical processors, program CMCI LVT entry, and query machine check banks that are shared by more 
than one logical processors. 


This section describes techniques system software can implement to manage CMCTI initialization, service CMCI 
interrupts in a efficient manner to minimize contentions to access shared MSR resources. 


15.5.2.1  CMCI Initialization 


Although a CMCI interrupt may be delivered to more than one logical processors depending on the nature of the 
corrected MC error, only one instance of the interrupt service routine needs to perform the necessary service and 
make queries to the machine-check banks. The following steps describes a technique that limits the amount of 
work the system has to do in response to a CMCI. 


° To provide maximum flexibility, system software should define per-thread data structure for each logical 
processor to allow equal-opportunity and efficient response to interrupt delivery. Specifically, the per-thread 
data structure should include a set of per-bank fields to track which machine check bank it needs to access in 
response to a delivered CMCI interrupt. The number of banks that needs to be tracked is determined by 
IA32_MCG_CAP[7:0]. 

¢ Initialization of per-thread data structure. The initialization of per-thread data structure must be done serially 
on each logical processor in the system. The sequencing order to start the per-thread initialization between 
different logical processor is arbitrary. But it must observe the following specific detail to satisfy the shared 
nature of specific MSR resources: 


a. Each thread initializes its data structure to indicate that it does not own any MC bank registers. 


b. Each thread examines IA32_MCi_CTL2[30] indicator for each bank to determine if another thread has 
already claimed ownership of that bank. 


¢ If IA32_MCi_CTL2[30] had been set by another thread. This thread can not own bank / and should 
proceed to step b. and examine the next machine check bank until all of the machine check banks are 
exhausted. 


¢ If IA32_MCi_CTL2[30] = 0, proceed to step c. 


c. Check whether writing a 1 into IA32_MCi_CTL2[30] can return with 1 on a subsequent read to determine 
this bank can support CMCI. 


¢ If IA32_MCi_CTL2[30] = 0, this bank does not support CMCI. This thread can not own bank / and should 
proceed to step b. and examine the next machine check bank until all of the machine check banks are 
exhausted. 


¢ If IA32_MCi_CTL2[30] = 1, modify the per-thread data structure to indicate this thread claims 
ownership to the MC bank; proceed to initialize the error threshold count (bits 15:0) of that bank as 
described in Chapter 15, “CMCI Threshold Management”. Then proceed to step b. and examine the next 
machine check bank until all of the machine check banks are exhausted. 


e ~=©After the thread has examined all of the machine check banks, it sees if it owns any MC banks to service CMCI. 
If any bank has been claimed by this thread: 


— Ensure that the CMCI interrupt handler has been set up as described in Chapter 15, “CMCI Interrupt 
Handler”. 


— Initialize the CMCI LVT entry, as described in Section 15.5.1, “CMCI Local APIC Interface”. 
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— Log and clear all of IA32_MCi_Status registers for the banks that this thread owns. This will allow new 
errors to be logged. 


15.5.2.2 CMCI Threshold Management 


The Corrected MC error threshold field, IA32_MCi_CTL2[14:0], is architecturally defined. Specifically, all these bits 
are writable by software, but different processor implementations may choose to implement less than 15 bits as 
threshold for the overflow comparison with IA32_MCi_STATUS[52:38]. The following describes techniques that 
software can manage CMCI threshold to be compatible with changes in implementation characteristics: 


¢ Software can set the initial threshold value to 1 by writing 1 to IA32_MCi_CTL2[14:0]. This will cause overflow 
condition on every corrected MC error and generates a CMCI interrupt. 


* To increase the threshold and reduce the frequency of CMCI servicing: 
a. Find the maximum threshold value a given processor implementation supports. The steps are: 
* Write 7FFFH to IA32_MCi_CTL2[14:0], 


¢ Read back IA32_MCi_CTL2[14:0]; these 15 bits (14:0) contain the maximum threshold supported by 
the processor. 


b. Increase the threshold to a value below the maximum value discovered using step a. 


15.5.2.3. CMCI Interrupt Handler 
The following describes techniques system software may consider to implement a CMCI service routine: 


¢ The service routine examines its private per-thread data structure to check which set of MC banks it has 
ownership. If the thread does not have ownership of a given MC bank, proceed to the next MC bank. Ownership 
is determined at initialization time which is described in Section 15.5.2.1. 


If the thread had claimed ownership to an MC bank, this technique will allow each logical processors to handle 
corrected MC errors independently and requires no synchronization to access shared MSR resources. Consult 
Example 15-5 for guidelines on logging when processing CMCI. 


15.6 RECOVERY OF UNCORRECTED RECOVERABLE (UCR) ERRORS 


Recovery of uncorrected recoverable machine check errors is an enhancement in machine-check architecture. The 
first processor that supports this feature is 45 nm Intel 64 processor on which CPUID reports 
DisplayFamily_DisplayModel as O6H_2EH (see CPUID instruction in Chapter 3, “Instruction Set Reference, A-L” in 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A). This allow system software to 
perform recovery action on certain class of uncorrected errors and continue execution. 


15.6.1 Detection of Software Error Recovery Support 


Software must use bit 24 of IA32_MCG_CAP (MCG_SER_P) to detect the presence of software error recovery 
support (see Figure 15-2). When IA32_MCG_CAP[24] is set, this indicates that the processor supports software 
error recovery. When this bit is clear, this indicates that there is no support for error recovery from the processor 
and the primary responsibility of the machine check handler is logging the machine check error information and 
shutting down the system. 


The new class of architectural MCA errors from which system software can attempt recovery is called Uncorrected 
Recoverable (UCR) Errors. UCR errors are uncorrected errors that have been detected and signaled but have not 
corrupted the processor context. For certain UCR errors, this means that once system software has performed a 
certain recovery action, it is possible to continue execution on this processor. UCR error reporting provides an error 
containment mechanism for data poisoning. The machine check handler will use the error log information from the 
error reporting registers to analyze and implement specific error recovery actions for UCR errors. 
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15.6.2 UCR Error Reporting and Logging 


IA32_MCi_STATUS MSR is used for reporting UCR errors and existing corrected or uncorrected errors. The defini- 
tions of IA32_MCi_STATUS, including bit fields to identify UCR errors, is shown in Figure 15-6. UCR errors can be 
signaled through either the corrected machine check interrupt (CMCI) or machine check exception (MCE) path 
depending on the type of the UCR error. 


When IA32_MCG_CAP[24] is set, a UCR error is indicated by the following bit settings in the IA32_MCi_STATUS 
register: 


* Valid (bit 63) = 1 
* UC (bit 61) = 1 
* PCC (bit 57) = 0 


Additional information from the IA32_MCi_MISC and the IA32_MCi_ADDR registers for the UCR error are available 
when the ADDRV and the MISCV flags in the IA32_MCi_STATUS register are set (See Section 15.3.2.4). The MCA 
error code field of the IA32_MCi_STATUS register indicates the type of UCR error. System software can interpret 
the MCA error code field to analyze and identify the necessary recovery action for the given UCR error. 


In addition, the IA32_MCi_STATUS register bit fields, bits 56:55, are defined (see Figure 15-6) to provide addi- 
tional information to help system software to properly identify the necessary recovery action for the UCR error: 


° S (Signaling) flag, bit 56 - Indicates (when set) that a machine check exception was generated for the UCR 
error reported in this MC bank and system software needs to check the AR flag and the MCA error code fields in 
the IA32_MCi_STATUS register to identify the necessary recovery action for this error. When the S flag in the 
IA32_MCi_STATUS register is clear, this UCR error was not signaled via a machine check exception and instead 
was reported as a corrected machine check (CMC). System software is not required to take any recovery action 
when the S flag in the IA32_MCi_STATUS register is clear. 


¢ AR (Action Required) flag, bit 55 - Indicates (when set) that MCA error code specific recovery action must be 
performed by system software at the time this error was signaled. This recovery action must be completed 
successfully before any additional work is scheduled for this processor. When the RIPV flag in the 
IA32_MCG_STATUS is clear, an alternative execution stream needs to be provided; when the MCA error code 
specific recovery specific recovery action cannot be successfully completed, system software must shut down 
the system. When the AR flag in the IA32_MCi_STATUS register is clear, system software may still take MCA 
error code specific recovery action but this is optional; system software can safely resume program execution 
at the instruction pointer saved on the stack from the machine check exception when the RIPV flag in the 
IA32_MCG_STATUS register is set. 


Both the S and the AR flags in the IA32_MCi_STATUS register are defined to be sticky bits, which mean that once 
set, the processor does not clear them. Only software and good power-on reset can clear the S and the AR-flags. 
Both the S and the AR flags are only set when the processor reports the UCR errors (MCG_CAP[24] is set). 


15.6.3. UCR Error Classification 
With the S and AR flag encoding in the IA32_MCi_STATUS register, UCR errors can be classified as: 


¢ Uncorrected no action required (UCNA) - is a UCR error that is not signaled via a machine check exception and, 
instead, is reported to system software as a corrected machine check error. UCNA errors indicate that some 
data in the system is corrupted, but the data has not been consumed and the processor state is valid and you 
may continue execution on this processor. UCNA errors require no action from system software to continue 
execution. A UCNA error is indicated with UC=1, PCC=0, S=0 and AR=0 in the IA32_MCi_STATUS register. 


° Software recoverable action optional (GRAO) - a UCR error is signaled either via a machine check exception or 
CMCI. System software recovery action is optional and not required to continue execution from this machine 
check exception. SRAO errors indicate that some data in the system is corrupt, but the data has not been 
consumed and the processor state is valid. SRAO errors provide the additional error information for system 
software to perform a recovery action. An SRAO error when signaled as a machine check is indicated with 
UC=1, PCC=0, S=1, EN=1 and AR=0 in the IA32_MCi_STATUS register. In cases when SRAO is signaled via 
CMCI the error signature is indicated via UC=1, PCC=0, S=0. Recovery actions for SRAO errors are MCA error 
code specific. The MISCV and the ADDRV flags in the IA32_MCi_STATUS register are set when the additional 
error information is available from the IA32_MCi_MISC and the IA32_MCi_ADDR registers. System software 
needs to inspect the MCA error code fields in the IA32_MCi_STATUS register to identify the specific recovery 
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action for a given SRAO error. If MISCV and ADDRV are not set, it is recommended that no system software 
error recovery be performed however, system software can resume execution. 


° Software recoverable action required (SRAR) - a UCR error that requires system software to take a recovery 
action on this processor before scheduling another stream of execution on this processor. SRAR errors indicate 
that the error was detected and raised at the point of the consumption in the execution flow. An SRAR error is 
indicated with UC=1, PCC=0, S=1, EN=1 and AR=1 in the IA32_MCi_STATUS register. Recovery actions are 
MCA error code specific. The MISCV and the ADDRV flags in the IA32_MCi_STATUS register are set when the 
additional error information is available from the IA32_MCi_MISC and the IA32_MCi_ADDR registers. System 
software needs to inspect the MCA error code fields in the IA32_MCi_STATUS register to identify the specific 
recovery action for a given SRAR error. If MISCV and ADDRV are not set, it is recommended that system 
software shutdown the system. 


Table 15-7 summarizes UCR, corrected, and uncorrected errors. 

Table 15-7. MC Error Classifications 
Type of Error’ UC |EN /|PCC |S _ |AR | Signaling | Software Action Example 
Uncorrected Error (UC) | 1 1 1 xX |X MCE If EN=1, reset the system, else log 
and OK to keep the system running. 
SRAR 1 1 0 1 41 MCE For known MCACOD, take specific Cache to processor load 
recovery action; error. 
For unknown MCACOD, must 
bugcheck. 


If OVER=1, reset system, else take 
specific recovery action. 


SRAO 1 x? 0 x2 10 MCE/CMC | For known MCACOD, take specific Patrol scrub and explicit 
recovery action; writeback poison errors. 


For unknown MCACOD, OK to keep 
the system running. 


UCNA 1 xX 0 0 |0 CMC Log the error and Ok to keep the Poison detection error. 
system running. 

Corrected Error (CE) 0 x X xX |X CMC Log the error and no corrective ECC in caches and 
action required. memory. 

NOTES: 


1. SRAR, SRAO and UCNA errors are supported by the processor only when IA32_MCG_CAP[24] (MCG_SER_P) is set. 
2. EN=1, S=1 when signaled via MCE. EN=x, S=O when signaled via CMC. 


15.6.4 UCR Error Overwrite Rules 


In general, the overwrite rules are as follows: 

¢ UCR errors will overwrite corrected errors. 

® Uncorrected (PCC=1) errors overwrite UCR (PCC=0) errors. 
¢ UCR errors are not written over previous UCR errors. 

* Corrected errors do not write over previous UCR errors. 


Regardless of whether the ist error is retained or the 2nd error is overwritten over the 1st error, the OVER flag in 
the IA32_MCi_STATUS register will be set to indicate an overflow condition. As the S flag and AR flag in the 
IA32_MCi_STATUS register are defined to be sticky flags, a second event cannot clear these 2 flags once set, 
however the MC bank information may be filled in for the 2nd error. The table below shows the overwrite rules and 
how to treat a second error if the first event is already logged in a MC bank along with the resulting bit setting of 
the UC, PCC, and AR flags in the IA32_MCi_STATUS register. As UCNA and SRAO errors do not require recovery 
action from system software to continue program execution, a system reset by system software is not required 
unless the AR flag or PCC flag is set for the UCR overflow case (OVER=1, VAL=1, UC=1, PCC=0). 
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Table 15-8 lists overwrite rules for uncorrected errors, corrected errors, and uncorrected recoverable errors. 
Table 15-8. Overwrite Rules for UC, CE, and UCR Errors 


First Event Second Event UC PCC |S AR MCA Bank Reset System 
ce UCR 1 0 O if UCNA, else 1 | 1 if SRAR, else O | second yes, if AR=1 
UCR CE 1 0 Oif UCNA, else 1 | 1if SRAR, else O | first yes, if AR=1 
UCNA UCNA 1 0 0 0 first no 

UCNA SRAO 1 0 1 0 first no 

UCNA SRAR 1 0 1 1 first yes 

SRAO UCNA 1 0 1 0 first no 

SRAO SRAO 1 0 1 0 first no 

SRAO SRAR 1 0 1 1 first yes 

SRAR UCNA 1 0 1 1 first yes 

SRAR SRAO 1 0 1 1 first yes 

SRAR SRAR 1 0 1 1 first yes 

UCR UC 1 1 undefined undefined second yes 

UC UCR 1 1 undefined undefined first yes 


15.7. MACHINE-CHECK AVAILABILITY 


The machine-check architecture and machine-check exception (#MC) are model-specific features. Software can 
execute the CPUID instruction to determine whether a processor implements these features. Following the execu- 
tion of the CPUID instruction, the settings of the MCA flag (bit 14) and MCE flag (bit 7) in EDX indicate whether the 
processor implements the machine-check architecture and machine-check exception. 


15.8 MACHINE-CHECK INITIALIZATION 


To use the processors machine-check architecture, software must initialize the processor to activate the machine- 
check exception and the error-reporting mechanism. 


Example 15-1 gives pseudocode for performing this initialization. This pseudocode checks for the existence of the 
machine-check architecture and exception; it then enables machine-check exception and the error-reporting 
register banks. The pseudocode shown is compatible with the Pentium 4, Intel Xeon, Intel Atom, P6 family, and 
Pentium processors. 


Following power up or power cycling, IA32_MCi_STATUS registers are not guaranteed to have valid data until after 
they are initially cleared to zero by software (as shown in the initialization pseudocode in Example 15-1). 


Example 15-1. Machine-Check Initialization Pseudocode 


Check CPUID Feature Flags for MCE and MCA support 
IF CPU supports MCE 
THEN 
IF CPU supports MCA 
THEN 
IF (IA32_MCG_CAP.MCG_CTL_P = 1) 
(* IA32_MCG_CTL register is present *) 
THEN 
IA32_MCG_CTL <— FFFFFFFFFFFFFFFFH; 
(* enables all MCA features *) 


Fl 


IF (IA32_MCG_CAP.MCG_LMCE_P = 1 and IA32_FEATURE_CONTROL.LOCK = 1 and IA32_FEATURE_CONTROL.LMCE_ENABLED = 1) 
(* IA32_MCG_EXT_CTL register is present and platform has enabled LMCE to permit system software to use LMCE *) 


Vol. 3B 15-19 


MACHINE-CHECK ARCHITECTURE 


Fl 


THEN 

IA32_MCG_EXT_CTL < IA32_MCG_EXT_CTL | 01H; 

(* System software enables LMCE capability for hardware to signal MCE to a single logical processor*) 
Fl 


(* Determine number of error-reporting banks supported *) 
COUNT< IA32_MCG_CAP.Count; 
MAX_BANK_NUMBER <— COUNT - 1; 


IF (Processor Family is 6H and Processor EXTMODEL:MODEL is less than 1AH) 
THEN 
(* Enable logging of all errors except for MCO_CTL register *) 
FOR error-reporting banks (1 through MAX_BANK_NUMBER) 
DO 
IA32_MCi_CTL < OFFFFFFFFFFFFFFFFH; 
OD 


ELSE 
(* Enable logging of all errors including MCO_CTL register *) 
FOR error-reporting banks (0 through MAX_BANK_NUMBER) 
DO 
IA32_MCi_CTL < OFFFFFFFFFFFFFFFFH; 
OD 
Fl 


(* BIOS clears all errors only on power-on reset *) 
IF (BIOS detects Power-on reset) 
THEN 
FOR error-reporting banks (0 through MAX_BANK_NUMBER) 
DO 
IA32_MCi_STATUS < 0; 
OD 
ELSE 
FOR error-reporting banks (0 through MAX_BANK_NUMBER) 
DO 
(Optional for BIOS and OS) Log valid errors 
(OS only) IA32_MCi_STATUS < 0; 
OD 


Fl 


Setup the Machine Check Exception (HMC) handler for vector 18 in IDT 


Set the MCE bit (bit 6) in CR4 register to enable Machine-Check Exceptions 


Fl 


15.9 INTERPRETING THE MCA ERROR CODES 


When the processor detects a machine-check error condition, it writes a 16-bit error code to the MCA error code 
field of one of the IA32_MCi_STATUS registers and sets the VAL (valid) flag in that register. The processor may also 
write a 16-bit model-specific error code in the IA32_MCi_STATUS register depending on the implementation of the 


machine-check architecture of the processor. 


The MCA error codes are architecturally defined for Intel 64 and IA-32 processors. To determine the cause of a 
machine-check exception, the machine-check exception handler must read the VAL flag for each 
IA32_MCi_STATUS register. If the flag is set, the machine check-exception handler must then read the MCA error 
code field of the register. It is the encoding of the MCA error code field [15:0] that determines the type of error 


being reported and not the register bank reporting it. 


There are two types of MCA error codes: simple error codes and compound error codes. 
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15.9.1. Simple Error Codes 


Table 15-9 shows the simple error codes. These unique codes indicate global error information. 


Table 15-9. 1A32_MCi_Status [15:0] Simple Error Code Encoding 


Error Code Binary Encoding Meaning 

No Error 0000 0000 0000 0000 No error has been reported to this bank of error-reporting 
registers. 

Unclassified 0000 0000 0000 0001 This error has not been classified into the MCA error classes. 

Microcode ROM Parity Error 0000 0000 0000 0010 Parity error in internal microcode ROM 

External Error 0000 0000 0000 0011 The BINIT# from another processor caused this processor to 
enter machine check.! 

FRC Error 0000 0000 0000 0100 FRC (functional redundancy check) main/secondary error. 

Internal Parity Error 0000 0000 0000 0101 Internal parity error. 

SMM Handler Code Access 0000 0000 0000 0110 An attempt was made by the SMM Handler to execute 

Violation outside the ranges specified by SMRR. 

Internal Timer Error 0000 0100 0000 0000 Internal timer error. 

1/0 Error 0000 1110 0000 1011 generic I/O error. 

Internal Unclassified OO00 01xx Xxxx XxXxx Internal unclassified errors. ¢ 

NOTES: 


1. BINIT# assertion will cause a machine check exception if the processor (or any processor on the same external bus) has BINIT# 
observation enabled during power-on configuration (hardware strapping) and if machine check exceptions are enabled (by setting 
CR4.MCE = 1). 


2. At least one X must equal one. Internal unclassified errors have not been classified. 


15.9.2 Compound Error Codes 


Compound error codes describe errors related to the TLBs, memory, caches, bus and interconnect logic, and 
internal timer. A set of sub-fields is common to all of compound errors. These sub-fields describe the type of 
access, level in the cache hierarchy, and type of request. Table 15-10 shows the general form of the compound 
error codes. 


Table 15-10. IA32_MCi_Status [15:0] Compound Error Code Encoding 
Type Form Interpretation 


Generic Cache Hierarchy 


OOOF 0000 0000 171LL 


Generic cache hierarchy error 


TLB Errors 


OOOF 0000 0001 TTLL 


(TT}TLB{LL}_ERR 


Memory Controller Errors 


OOOF 0000 1MMM CCCC 


{MMM}_CHANNEL{CCCC}_ERR 


Cache Hierarchy Errors 


OOOF 0001 RRRR TTLL 


{TT}CACHE{LL} {RRRR}_ERR 


Extended Memory Errors 


OOOF 0010 1MMM CCCC 


{MMM}_CHANNEL{CCCC}_ERR 


Bus and Interconnect Errors 


OOOF 1PPT RRRR IILL 


BUS{LL_{PP}_{RRRR}_{Il]_{T}_ERR 


The “Interpretation” column in the table indicates the name of a compound error. The name is constructed by 
substituting mnemonics for the sub-field names given within curly braces. For example, the error code 
ICACHEL1_RD_ERR is constructed from the form: 

{TT}CACHE{LL}{RRRR}_ERR, 

where {TT} is replaced by |, {LL} is replaced by L1, and {RRRR} is replaced by RD. 
For more information on the “Form” and “Interpretation” columns, see Sections Section 15.9.2.1, “Correction 
Report Filtering (F) Bit” through Section 15.9.2.5, “Bus and Interconnect Errors”. 
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15.9.2.1 Correction Report Filtering (F) Bit 
Starting with Intel Core Duo processors, bit 12 in the “Form” column in Table 15-10 is used to indicate that a partic- 
ular posting to a log may be the last posting for corrections in that line/entry, at least for some time: 


* Qin bit 12 indicates “normal” filtering (original P6/Pentium4/Atom/Xeon processor meaning). 


* 1in bit 12 indicates “corrected” filtering (filtering is activated for the line/entry in the posting). Filtering means 
that some or all of the subsequent corrections to this entry (in this structure) will not be posted. The enhanced 
error reporting introduced with the Intel Core Duo processors is based on tracking the lines affected by 
repeated corrections (see Section 15.4, “Enhanced Cache Error reporting”). This capability is indicated by 
IA32_MCG_CAP[11]. Only the first few correction events for a line are posted; subsequent redundant 
correction events to the same line are not posted. Uncorrected events are always posted. 


The behavior of error filtering after crossing the yellow threshold is model-specific. Filtering has meaning only for 
corrected errors (UC=0 in IA32_MCi_STATUS MSR). System software must ignore filtering bit (12) for uncorrected 


errors. 


15.9.2.2 Transaction Type (TT) Sub-Field 


The 2-bit TT sub-field (Table 15-11) indicates the type of transaction (data, instruction, or generic). The sub-field 
applies to the TLB, cache, and interconnect error conditions. Note that interconnect error conditions are primarily 
associated with P6 family and Pentium processors, which utilize an external APIC bus separate from the system 
bus. The generic type is reported when the processor cannot determine the transaction type. 


Table 15-11. Encoding for TT (Transaction Type) Sub-Field 


Binary Encoding 


15.9.2.3 Level (LL) Sub-Field 


Transaction Type Mnemonic 
Instruction I 00 
Data D 01 
Generic G 10 


The 2-bit LL sub-field (see Table 15-12) indicates the level in the memory hierarchy where the error occurred (level 
0, level 1, level 2, or generic). The LL sub-field also applies to the TLB, cache, and interconnect error conditions. 
The Pentium 4, Intel Xeon, Intel Atom, and P6 family processors support two levels in the cache hierarchy and one 
level in the TLBs. Again, the generic type is reported when the processor cannot determine the hierarchy level. 


Table 15-12. Level Encoding for LL (Memory Hierarchy Level) Sub-Field 


Hierarchy Level Mnemonic Binary Encoding 
Level 0 LO 00 
Level 1 1 01 
Level 2 L2 10 
Generic LG 11 


15.9.2.4 Request (RRRR) Sub-Field 


The 4-bit RRRR sub-field (see Table 15-13) indicates the type of action associated with the error. Actions include 
read and write operations, prefetches, cache evictions, and snoops. Generic error is returned when the type of 
error cannot be determined. Generic read and generic write are returned when the processor cannot determine the 
type of instruction or data request that caused the error. Eviction and snoop requests apply only to the caches. All 


of the other requests apply to TLBs, caches and interconnects. 


15-22 Vol. 3B 


MACHINE-CHECK ARCHITECTURE 


Table 15-13. Encoding of Request (RRRR) Sub-Field 


Request Type Mnemonic Binary Encoding 
Generic Error ERR 0000 
Generic Read RD 0001 
Generic Write WR 0010 
Data Read DRD 0011 
Data Write DWR 0100 
Instruction Fetch IRD 0101 
Prefetch PREFETCH 0110 
Eviction EVICT 0111 
Snoop SNOOP 1000 


15.9.2.5 Bus and Interconnect Errors 


The bus and interconnect errors are defined with the 2-bit PP (participation), 1-bit T (time-out), and 2-bit II 
(memory or I/O) sub-fields, in addition to the LL and RRRR sub-fields (see Table 15-14). The bus error conditions 
are implementation dependent and related to the type of bus implemented by the processor. Likewise, the inter- 
connect error conditions are predicated on a specific implementation-dependent interconnect model that describes 
the connections between the different levels of the storage hierarchy. The type of bus is implementation depen- 
dent, and as such is not specified in this document. A bus or interconnect transaction consists of a request involving 
an address and a response. 


Table 15-14. Encodings of PP, T, and II Sub-Fields 


Sub-Field Transaction Mnemonic Binary Encoding 
PP (Participation) Local processor* originated request SRC 00 
Local processor* responded to request RES 01 
Local processor* observed error as third party OBS 10 
Generic 11 
T (Time-out) Request timed out TIMEOUT 1 
Request did not time out NOTIMEOUT 0) 
Il (Memory or I/O) Memory Access M 00 
Reserved 01 
1/0 10 10 
Other transaction 11 


NOTE: 


* Local processor differentiates the processor reporting the error from other system components (including the APIC, other proces- 


sors, etc.). 
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15.9.2.6 Memory Controller and Extended Memory Errors 


The memory controller errors are defined with the 3-bit MMM (memory transaction type), and 4-bit CCCC (channel) 
sub-fields. The encodings for MMM and CCCC are defined in Table 15-15. Extended Memory errors use the same 
encodings and are used to report errors in memory used as a cache. 


Table 15-15. Encodings of MMM and CCCC Sub-Fields 


Sub-Field Transaction Mnemonic Binary Encoding 
MMM Generic undefined request GEN 000 
Memory read error RD 001 
Memory write error WR 010 
Address/Command Error AC 011 
Memory Scrubbing Error MS 100 
Reserved 101-111 
Ccccc Channel number CHN 0000-1110 
Channel not specified 1111 


Note that the CCCC channel number may be enumerated from zero separately by each memory controller on a 
system. On a multi-socket system, or a system with multiple memory controllers per socket, it is necessary to also 
consider which machine check bank logged the error. See Chapter 16 for details on specific implementations. 


15.9.3 Architecturally Defined UCR Errors 


Software recoverable compound error code are defined in this section. 


15.9.3.1 Architecturally Defined SRAO Errors 

The following two SRAO errors are architecturally defined. 

e UCR Errors detected by memory controller scrubbing; and 

¢ UCR Errors detected during L3 cache (L3) explicit writebacks. 


The MCA error code encodings for these two architecturally-defined UCR errors corresponds to sub-classes of 
compound MCA error codes (see Table 15-10). Their values and compound encoding format are given in Table 
15-16. 


Table 15-16. MCA Compound Error Code Encoding for SRAO Errors 


Type MCACOD Value | MCA Error Code Encoding’ 


Memory Scrubbing | COH - CFH 0000_0000_1100_CCCC 

OOOF 0000 1MMM CCCC (Memory Controller Error), where 
Memory subfield MMM = 100B (memory scrubbing) 
Channel subfield CCCC = channel # or generic 


L3 Explicit Writeback | 17AH 0000_0001_0111_1010 

OOOF 0001 RRRR TTLL (Cache Hierarchy Error) where 
Request subfields RRRR = 01118B (Eviction) 
Transaction Type subfields TT = 10B (Generic) 

Level subfields LL = 10B 


NOTES: 
1. Note that for both of these errors the correction report filtering (F) bit (bit 12) of the MCA error must be ignored. 
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Table 15-17 lists values of relevant bit fields of IA32_MCi_STATUS for architecturally defined SRAO errors. 
Table 15-17. IA32_MCi_STATUS Values for SRAO Errors 


SRAO Error Valid OVER uc [EN |MISCV ADDRV Pcc |S |AR |MCACOD 
Memory Scrubbing 1 0 1 x | 1 0 x' 10 | COH-CFH 
L3 Explicit Writeback 1 0 1 x? 14 1 0 x’ [0 17AH 
NOTES: 


1. When signaled as MCE, EN=1 and S=1. If error was signaled via CMC, then EN=x, and S=0. 


For both the memory scrubbing and L3 explicit writeback errors, the ADDRV and MISCV flags in the 
IA32_MCi_STATUS register are set to indicate that the offending physical address information is available from the 
IA32_MCi_MISC and the IA32_MCi_ADDR registers. For the memory scrubbing and L3 explicit writeback errors, 
the address mode in the IA32_MCi_MISC register should be set as physical address mode (010b) and the address 
LSB information in the IA32_MCi_MISC register should indicate the lowest valid address bit in the address informa- 
tion provided from the IA32_MCi_ADDR register. 


MCE signal is broadcast to all logical processors as outlined in Section 15.10.4.1. If LMCE is supported and enabled, 
some errors (not limited to UCR errors) may be delivered to only a single logical processor. System software should 
consult IA32_MCG_STATUS.LMCE_S to determine if the MCE signaled is only to this logical processor. 
IA32_MCi_STATUS banks can be shared by logical processors within a core or within the same package. So several 
logical processors may find an SRAO error in the shared IA32_MCi_STATUS bank but other processors do not find 
it in any of the IA32_MCi_STATUS banks. Table 15-18 shows the RIPV and EIPV flag indication in the 
IA32_MCG_STATUS register for the memory scrubbing and L3 explicit writeback errors on both the reporting and 
non-reporting logical processors. 


Table 15-18. IA32_MCG_STATUS Flag Indication for SRAO Errors 


SRAO Type Reporting Logical Processors Non-reporting Logical Processors 
RIPV EIPV RIPV EIPV 

Memory Scrubbing 1 0) 1 0 

L3 Explicit Writeback 1 0) 1 0 


15.9.3.2 Architecturally Defined SRAR Errors 

The following two SRAR errors are architecturally defined. 
¢ UCR Errors detected on data load; and 

¢ UCR Errors detected on instruction fetch. 


The MCA error code encodings for these two architecturally-defined UCR errors corresponds to sub-classes of 
compound MCA error codes (see Table 15-10). Their values and compound encoding format are given in Table 
15-19. 


Table 15-19. MCA Compound Error Code Encoding for SRAR Errors 
Type MCACOD Value | MCA Error Code Encoding’ 
Data Load 134H 0000_0001_0011_0100 
OOOF 0001 RRRR TTLL (Cache Hierarchy Error), where 
Request subfield RRRR = 0011B (Data Load) 
Transaction Type subfield TT= 01B (Data) 
Level subfield LL = OOB (Level 0) 
Instruction Fetch 150H 0000_0001_0101_0000 
OOOF 0001 RRRR TTLL (Cache Hierarchy Error), where 
Request subfield RRRR = 0101B (Instruction Fetch) 
Transaction Type subfield TT= OOB (Instruction) 
Level subfield LL = OOB (Level 0) 
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NOTES: 
1, Note that for both of these errors the correction report filtering (F) bit (bit 12) of the MCA error must be ignored. 


Table 15-20 lists values of relevant bit fields of IA32_MCi_STATUS for architecturally defined SRAR errors. 
Table 15-20. IA32_MCi_STATUS Values for SRAR Errors 


SRAR Error Valid OVER UC |EN_ |MISCV ADDRV PCC S |AR_ |MCACOD 
Data Load 1 0 1 1 1 1 0 1 1 134H 
Instruction Fetch 1 0 1 1 1 1 0 1 1 150H 


For both the data load and instruction fetch errors, the ADDRV and MISCV flags in the IA32_MCi_STATUS register 
are set to indicate that the offending physical address information is available from the IA32_MCi_MISC and the 
IA32_MCi_ADDR registers. For the memory scrubbing and L3 explicit writeback errors, the address mode in the 
IA32_MCi_MISC register should be set as physical address mode (010b) and the address LSB information in the 
IA32_MCi_MISC register should indicate the lowest valid address bit in the address information provided from the 
IA32_MCi_ADDR register. 


MCE signal is broadcast to all logical processors on the system on which the UCR errors are supported, except when 
the processor supports LMCE and LMCE is enabled by system software (see Section 15.3.1.5). The 
IA32_MCG_STATUS MSR allows system software to distinguish the affected logical processor of an SRAR error 
amongst logical processors that observed SRAR via MCi_STATUS bank. 


Table 15-21 shows the RIPV and EIPV flag indication in the IA32_MCG_STATUS register for the data load and 
instruction fetch errors on both the reporting and non-reporting logical processors. The recoverable SRAR error 
reported by a processor may be continuable, where the system software can interpret the context of continuable 
as follows: the error was isolated, contained. If software can rectify the error condition in the current instruction 
stream, the execution context on that logical processor can be continued without loss of information. 


Table 15-21. 1A32_MCG_STATUS Flag Indication for SRAR Errors 


SRAR Type Affected Logical Processor Non-Affected Logical Processors 

RIPV EIPV Continuable RIPV EIPV Continuable 
Recoverable- 1 1 Yes! 
continuable 
Recoverable-not- 0 x No 1 0 Yes 
continuable 
NOTES: 


1. see the definition of the context of “continuable” above and additional detail below. 


SRAR Error And Affected Logical Processors 

The affected logical processor is the one that has detected and raised an SRAR error at the point of the consump- 
tion in the execution flow. The affected logical processor should find the Data Load or the Instruction Fetch error 
information in the IA32_MCi_STATUS register that is reporting the SRAR error. 


Table 15-21 list the actionable scenarios that system software can respond to an SRAR error on an affected logical 
processor according to RIPV and EIPV values: 


¢ Recoverable-Continuable SRAR Error (RIPV=1, EIPV=1): 


For Recoverable-Continuable SRAR errors, the affected logical processor should find that both the 
IA32_MCG_STATUS.RIPV and the IA32_MCG_STATUS.EIPV flags are set, indicating that system software may 
be able to restart execution from the interrupted context if it is able to rectify the error condition. If system 
software cannot rectify the error condition then it must treat the error as a recoverable error where restarting 
execution with the interrupted context is not possible. Restarting without rectifying the error condition will 
result in most cases with another SRAR error on the same instruction. 
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¢ Recoverable-not-continuable SRAR Error (RIPV=0, EIPV=x): 
For Recoverable-not-continuable errors, the affected logical processor should find that either 
— JA32_MCG_STATUS.RIPV= 0, IA32_MCG_STATUS.EIPV=1, or 
— JIA32_MCG_STATUS.RIPV= 0, IA32_MCG_STATUS.EIPV=0. 


In either case, this indicates that the error is detected at the instruction pointer saved on the stack for this 
machine check exception and restarting execution with the interrupted context is not possible. System 
software may take the following recovery actions for the affected logical processor: 


¢ The current executing thread cannot be continued. System software must terminate the interrupted 
stream of execution and provide a new stream of execution on return from the machine check handler 
for the affected logical processor. 


SRAR Error And Non-Affected Logical Processors 


The logical processors that observed but not affected by an SRAR error should find that the RIPV flag in the 
IA32_MCG_STATUS register is set and the EIPV flag in the IA32_MCG_STATUS register is cleared, indicating that it 
is safe to restart the execution at the instruction saved on the stack for the machine check exception on these 
processors after the recovery action is successfully taken by system software. 


15.9.4 Multiple MCA Errors 


When multiple MCA errors are detected within a certain detection window, the processor may aggregate the 
reporting of these errors together as a single event, i.e. a single machine exception condition. If this occurs, 
system software may find multiple MCA errors logged in different MC banks on one logical processor or find 
multiple MCA errors logged across different processors for a single machine check broadcast event. In order to 
handle multiple UCR errors reported from a single machine check event and possibly recover from multiple errors, 
system software may consider the following: 


¢ Whether it can recover from multiple errors is determined by the most severe error reported on the system. If 
the most severe error is found to be an unrecoverable error (VAL=1, UC=1, PCC=1 and EN=1) after system 
software examines the MC banks of all processors to which the MCA signal is broadcast, recovery from the 
multiple errors is not possible and system software needs to reset the system. 


¢ When multiple recoverable errors are reported and no other fatal condition (e.g. overflowed condition for SRAR 
error) is found for the reported recoverable errors, it is possible for system software to recover from the 
multiple recoverable errors by taking necessary recovery action for each individual recoverable error. However, 
system software can no longer expect one to one relationship with the error information recorded in the 
IA32_MCi_STATUS register and the states of the RIPV and EIPV flags in the IA32_MCG_STATUS register as the 
states of the RIPV and the EIPV flags in the IA32_MCG_STATUS register may indicate the information for the 
most severe error recorded on the processor. System software is required to use the RIPV flag indication in the 
IA32_MCG_STATUS register to make a final decision of recoverability of the errors and find the restart-ability 
requirement after examining each IA32_MCi_STATUS register error information in the MC banks. 


In certain cases where system software observes more than one SRAR error logged for a single logical 
processor, it can no longer rely on affected threads as specified in Table 15-20 above. System software is 
recommended to reset the system if this condition is observed. 


15.9.5 Machine-Check Error Codes Interpretation 


Chapter 16, “Interpreting Machine-Check Error Codes,” provides information on interpreting the MCA error code, 
model-specific error code, and other information error code fields. For P6 family processors, information has been 
included on decoding external bus errors. For Pentium 4 and Intel Xeon processors; information is included on 
external bus, internal timer and cache hierarchy errors. 
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15.10 GUIDELINES FOR WRITING MACHINE-CHECK SOFTWARE 


The machine-check architecture and error logging can be used in three different ways: 
* To detect machine errors during normal instruction execution, using the machine-check exception (#MC). 
¢ To periodically check and log machine errors. 


° To examine recoverable UCR errors, determine software recoverability and perform recovery actions via a 
machine-check exception handler or a corrected machine-check interrupt handler. 


To use the machine-check exception, the operating system or executive software must provide a machine-check 
exception handler. This handler may need to be designed specifically for each family of processors. 


A special program or utility is required to log machine errors. 


Guidelines for writing a machine-check exception handler or a machine-error logging utility are given in the 
following sections. 


15.10.1  Machine-Check Exception Handler 


The machine-check exception (#MC) corresponds to vector 18. To service machine-check exceptions, a trap gate 
must be added to the IDT. The pointer in the trap gate must point to a machine-check exception handler. Two 
approaches can be taken to designing the exception handler: 


1. The handler can merely log all the machine status and error information, then call a debugger or shut down the 
system. 


2. The handler can analyze the reported error information and, in some cases, attempt to correct the error and 
restart the processor. 


For Pentium 4, Intel Xeon, Intel Atom, P6 family, and Pentium processors; virtually all machine-check conditions 
cannot be corrected (they result in abort-type exceptions). The logging of status and error information is therefore 
a baseline implementation requirement. 


When IA32_MCG_CAP[24] is clear, consider the following when writing a machine-check exception handler: 


* To determine the nature of the error, the handler must read each of the error-reporting register banks. The 
count field in the IA32_MCG_CAP register gives number of register banks. The first register of register bank 0 
is at address 400H. 


¢ The VAL (valid) flag in each IA32_MCi_STATUS register indicates whether the error information in the register 
is valid. If this flag is clear, the registers in that bank do not contain valid error information and do not need to 
be checked. 


¢ To write a portable exception handler, only the MCA error code field in the IA32_MCi_STATUS register should be 
checked. See Section 15.9, “Interpreting the MCA Error Codes,” for information that can be used to write an 
algorithm to interpret this field. 


* Correctable errors are corrected automatically by the processor. The UC flag in each IA32_MCi_STATUS reg- 
ister indicates whether the processor automatically corrected an error. 


° The RIPV, PCC, and OVER flags in each IA32_MCi_STATUS register indicate whether recovery from the error is 
possible. If PCC or OVER are set, recovery is not possible. If RIPV is not set, program execution can not be 
restarted reliably. When recovery is not possible, the handler typically records the error information and signals 
an abort to the operating system. 


¢ The RIPV flag in the IA32_MCG_STATUS register indicates whether the program can be restarted at the 
instruction indicated by the instruction pointer (the address of the instruction pushed on the stack when the 
exception was generated). If this flag is clear, the processor may still be able to be restarted (for debugging 
purposes) but not without loss of program continuity. 


° For unrecoverable errors, the EIPV flag in the IA32_MCG_STATUS register indicates whether the instruction 
indicated by the instruction pointer pushed on the stack (when the exception was generated) is related to the 
error. If the flag is clear, the pushed instruction may not be related to the error. 


¢ The MCIP flag in the IA32_MCG_STATUS register indicates whether a machine-check exception was generated. 
Before returning from the machine-check exception handler, software should clear this flag so that it can be 
used reliably by an error logging utility. The MCIP flag also detects recursion. The machine-check architecture 
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does not support recursion. When the processor detects machine-check recursion, it enters the shutdown 
state. 


Example 15-2 gives typical steps carried out by a machine-check exception handler. 


Example 15-2. Machine-Check Exception Handler Pseudocode 


IF CPU supports MCE 
THEN 
IF CPU supports MCA 
THEN 
call errorlogging routine; (* returns restartability *) 
Fi; 
ELSE (* Pentium(R) processor compatible *) 
READ P5_MC_ADDR 
READ P5_MC_TYPE; 
report RESTARTABILITY to console; 
FI; 
IF error is not restartable 
THEN 
report RESTARTABILITY to console; 
abort system; 
Fi; 
CLEAR MCIP flag in |A32_MCG_STATUS; 


15.10.2 Pentium Processor Machine-Check Exception Handling 


Machine-check exception handler on P6 family, Intel Atom and later processor families, should follow the guidelines 
described in Section 15.10.1 and Example 15-2 that check the processor’s support of MCA. 


NOTE 


On processors that support MCA (CPUID.1.EDX.MCA = 1) reading the P5_MC_TYPE and 
P5_MC_ADDR registers may produce invalid data. 


When machine-check exceptions are enabled for the Pentium processor (MCE flag is set in control register CR4), 
the machine-check exception handler uses the RDMSR instruction to read the error type from the PS5_MC_TYPE 
register and the machine check address from the P5_MC_ADDR register. The handler then normally reports these 
register values to the system console before aborting execution (See Example 15-2). 


15.10.3 Logging Correctable Machine-Check Errors 


The error handling routine for servicing the machine-check exceptions is responsible for logging uncorrected 
errors. 


If a machine-check error is correctable, the processor does not generate a machine-check exception for it. To 
detect correctable machine-check errors, a utility program must be written that reads each of the machine-check 
error-reporting register banks and logs the results in an accounting file or data structure. This utility can be imple- 
mented in either of the following ways. 


¢ Asystem daemon that polls the register banks on an infrequent basis, such as hourly or daily. 


° Auser-initiated application that polls the register banks and records the exceptions. Here, the actual polling 
service is provided by an operating-system driver or through the system call interface. 


¢ = Aninterrupt service routine servicing CMCI can read the MC banks and log the error. Please refer to Section 
15.10.4.2 for guidelines on logging correctable machine checks. 


Example 15-3 gives pseudocode for an error logging utility. 
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Example 15-3. Machine-Check Error Logging Pseudocode 


Assume that execution is restartable; 
IF the processor supports MCA 
THEN 
FOR each bank of machine-check registers 
DO 
READ IA32_MCi_STATUS; 
IF VAL flag in IA32_MCi_STATUS = 1 
THEN 
IF ADDRV flag in IA32_MCi_STATUS = 1 
THEN READ IA32_MCi_ADDR; 
FI; 
IF MISCV flag in IA32_MCi_STATUS = 1 
THEN READ IA32_MCi_MISC; 
Fl; 
IF MCIP flag in |A32_MCG_STATUS = 1 
(* Machine-check exception is in progress *) 
AND PCC flag in IA32_MCi_STATUS = 1 
OR RIPV flag in IA32_MCG_STATUS = 0 
(* execution is not restartable *) 
THEN 
RESTARTABILITY = FALSE; 
return RESTARTABILITY to calling procedure; 
FI; 
Save time-stamp counter and processor ID; 
Set [A32_MCi_STATUS to all Os; 
Execute serializing instruction (i.e., CPUID); 
FI; 
OD; 
Fl; 
If the processor supports the machine-check architecture, the utility reads through the banks of error-reporting 
registers looking for valid register entries. It then saves the values of the IA32_MCi_STATUS, IA32_MCi_ADDR, 
IA32_MCi_MISC and IA32_MCG_STATUS registers for each bank that is valid. The routine minimizes processing 
time by recording the raw data into a system data structure or file, reducing the overhead associated with polling. 
User utilities analyze the collected data in an off-line environment. 


When the MCIP flag is set in the IA32_MCG_STATUS register, a machine-check exception is in progress and the 
machine-check exception handler has called the exception logging routine. 


Once the logging process has been completed the exception-handling routine must determine whether execution 
can be restarted, which is usually possible when damage has not occurred (The PCC flag is clear, in the 
IA32_MCi_STATUS register) and when the processor can guarantee that execution is restartable (the RIPV flag is 
set in the IA32_MCG_STATUS register). If execution cannot be restarted, the system is not recoverable and the 
exception-handling routine should signal the console appropriately before returning the error status to the Oper- 
ating System kernel for subsequent shutdown. 


The machine-check architecture allows buffering of exceptions from a given error-reporting bank although the 
Pentium 4, Intel Xeon, Intel Atom, and P6 family processors do not implement this feature. The error logging 
routine should provide compatibility with future processors by reading each hardware error-reporting bank's 
IA32_MCi_STATUS register and then writing Os to clear the OVER and VAL flags in this register. The error logging 
utility should re-read the IA32_MCi_STATUS register for the bank ensuring that the valid bit is clear. The processor 
will write the next error into the register bank and set the VAL flags. 


Additional information that should be stored by the exception-logging routine includes the processor’s time-stamp 
counter value, which provides a mechanism to indicate the frequency of exceptions. A multiprocessing operating 
system stores the identity of the processor node incurring the exception using a unique identifier, such as the 
processor's APIC ID (see Section 10.8, “Handling Interrupts”). 


The basic algorithm given in Example 15-3 can be modified to provide more robust recovery techniques. For 
example, software has the flexibility to attempt recovery using information unavailable to the hardware. Specifi- 
cally, the machine-check exception handler can, after logging carefully analyze the error-reporting registers when 
the error-logging routine reports an error that does not allow execution to be restarted. These recovery techniques 
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can use external bus related model-specific information provided with the error report to localize the source of the 
error within the system and determine the appropriate recovery strategy. 


15.10.4 Machine-Check Software Handler Guidelines for Error Recovery 


15.10.4.1 Machine-Check Exception Handler for Error Recovery 


When writing a machine-check exception (MCE) handler to support software recovery from Uncorrected Recover- 
able (UCR) errors, consider the following: 


When IA32_MCG_CAP [24] is zero, there are no recoverable errors supported and all machine-check are fatal 
exceptions. The logging of status and error information is therefore a baseline implementation requirement. 


When IA32_MCG_CAP [24] is 1, certain uncorrected errors called uncorrected recoverable (UCR) errors may be 
software recoverable. The handler can analyze the reported error information, and in some cases attempt to 
recover from the uncorrected error and continue execution. 


For processors on which CPUID reports DisplayFamily_DisplayModel as O6H_OEH and onward, an MCA signal is 
broadcast to all logical processors in the system (see CPUID instruction in Chapter 3, “Instruction Set 
Reference, A-L” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A). Due to the 
potentially shared machine check MSR resources among the logical processors on the same package/core, the 
MCE handler may be required to synchronize with the other processors that received a machine check error and 
serialize access to the machine check registers when analyzing, logging and clearing the information in the 
machine check registers. 


— On processors that indicate ability for local machine-check exception (MCG_LMCE_P), hardware can choose 
to report the error to only a single logical processor if system software has enabled LMCE by setting 
IA32_MCG_EXT_CTL[LMCE_EN] = 1 as outlined in Section 15.3.1.5. 


The VAL (valid) flag in each IA32_MCi_STATUS register indicates whether the error information in the register 
is valid. If this flag is clear, the registers in that bank do not contain valid error information and should not be 
checked. 


The MCE handler is primarily responsible for processing uncorrected errors. The UC flag in each 
IA32_MCi_Status register indicates whether the reported error was corrected (UC=0) or uncorrected (UC=1). 
The MCE handler can optionally log and clear the corrected errors in the MC banks if it can implement software 
algorithm to avoid the undesired race conditions with the CMCI or CMC polling handler. 


For uncorrectable errors, the EIPV flag in the IA32_MCG_STATUS register indicates (when set) that the 
instruction pointed to by the instruction pointer pushed onto the stack when the machine-check exception is 
generated is directly associated with the error. When this flag is cleared, the instruction pointed to may not be 
associated with the error. 


The MCIP flag in the IA32_MCG_STATUS register indicates whether a machine-check exception was generated. 
When a machine check exception is generated, it is expected that the MCIP flag in the IA32_MCG_STATUS 
register is set to 1. If it is not set, this machine check was generated by either an INT 18 instruction or some 
piece of hardware signaling an interrupt with vector 18. 


When IA32_MCG_CAP [24] is 1, the following rules can apply when writing a machine check exception (MCE) 
handler to support software recovery: 


The PCC flag in each IA32_MCi_STATUS register indicates whether recovery from the error is possible for 
uncorrected errors (UC=1). If the PCC flag is set for enabled uncorrected errors (UC=1 and EN=1), recovery is 
not possible. When recovery is not possible, the MCE handler typically records the error information and signals 
the operating system to reset the system. 


The RIPV flag in the IA32_MCG_STATUS register indicates whether restarting the program execution from the 
instruction pointer saved on the stack for the machine check exception is possible. When the RIPV is set, 
program execution can be restarted reliably when recovery is possible. If the RIPV flag is not set, program 
execution cannot be restarted reliably. In this case the recovery algorithm may involve terminating the current 
program execution and resuming an alternate thread of execution upon return from the machine check handler 
when recovery is possible. When recovery is not possible, the MCE handler signals the operating system to 
reset the system. 
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¢ When the EN flag is zero but the VAL and UC flags are one in the IA32_MCi_STATUS register, the reported 
uncorrected error in this bank is not enabled. As uncorrected errors with the EN flag = 0 are not the source of 
machine check exceptions, the MCE handler should log and clear non-enabled errors when the S bit is set and 
should continue searching for enabled errors from the other IA32_MCi_STATUS registers. Note that when 
IA32_MCG_CAP [24] is 0, any uncorrected error condition (VAL =1 and UC=1) including the one with the EN 
flag cleared are fatal and the handler must signal the operating system to reset the system. For the errors that 
do not generate machine check exceptions, the EN flag has no meaning. 


¢ When the VAL flag is one, the UC flag is one, the EN flag is one and the PCC flag is zero in the IA32_MCi_STATUS 
register, the error in this bank is an uncorrected recoverable (UCR) error. The MCE handler needs to examine 
the S flag and the AR flag to find the type of the UCR error for software recovery and determine if software error 
recovery is possible. 


¢ When both the S and the AR flags are clear in the IA32_MCi_STATUS register for the UCR error (VAL=1, UC=1, 
EN=x and PCC=0), the error in this bank is an uncorrected no-action required error (UCNA). UCNA errors are 
uncorrected but do not require any OS recovery action to continue execution. These errors indicate that some 
data in the system is corrupt, but that data has not been consumed and may not be consumed. If that data is 
consumed a non-UCNA machine check exception will be generated. UCNA errors are signaled in the same way 
as corrected machine check errors and the CMCI and CMC polling handler is primarily responsible for handling 
UCNA errors. Like corrected errors, the MCA handler can optionally log and clear UCNA errors as long as it can 
avoid the undesired race condition with the CMCI or CMC polling handler. As UCNA errors are not the source of 
machine check exceptions, the MCA handler should continue searching for uncorrected or software recoverable 
errors in all other MC banks. 


¢ When the S flag in the IA32_MCi_STATUS register is set for the UCR error ((VAL=1, UC=1, EN=1 and PCC=0), 
the error in this bank is software recoverable and it was signaled through a machine-check exception. The AR 
flag in the IA32_MCi_STATUS register further clarifies the type of the software recoverable errors. 


¢ When the AR flag in the IA32_MCi_STATUS register is clear for the software recoverable error (VAL=1, UC=1, 
EN=1, PCC=0 and S=1), the error in this bank is a software recoverable action optional (SRAO) error. The MCE 
handler and the operating system can analyze the IA32_MCi_STATUS [15:0] to implement MCA error code 
specific optional recovery action, but this recovery action is optional. System software can resume the program 
execution from the instruction pointer saved on the stack for the machine check exception when the RIPV flag 
in the IA32_MCG_STATUS register is set. 


° Even if the OVER flag in the IA32_MCi_STATUS register is set for the SRAO error (VAL=1, UC=1, EN=1, PCC=0, 
S=1 and AR=0), the MCE handler can take recovery action for the SRAO error logged in the IA32_MCi_STATUS 
register. Since the recovery action for SRAO errors is optional, restarting the program execution from the 
instruction pointer saved on the stack for the machine check exception is still possible for the overflowed SRAO 
error if the RIPV flag in the IA32_MCG_STATUS is set. 


¢ When the AR flag in the IA32_MCi_STATUS register is set for the software recoverable error (VAL=1, UC=1, 
EN=1, PCC=0 and S=1), the error in this bank is a software recoverable action required (SRAR) error. The MCE 
handler and the operating system must take recovery action in order to continue execution after the machine- 
check exception. The MCA handler and the operating system need to analyze the IA32_MCi_STATUS [15:0] to 
determine the MCA error code specific recovery action. If no recovery action can be performed, the operating 
system must reset the system. 


¢ When the OVER flag in the IA32_MCi_STATUS register is set for the SRAR error (VAL=1, UC=1, EN=1, PCC=0, 
S=1 and AR=1), the MCE handler cannot take recovery action as the information of the SRAR error in the 
IA32_MCi_STATUS register was potentially lost due to the overflow condition. Since the recovery action for 
SRAR errors must be taken, the MCE handler must signal the operating system to reset the system. 

¢ When the MCE handler cannot find any uncorrected (VAL=1, UC=1 and EN=1) or any software recoverable 
errors (VAL=1, UC=1, EN=1, PCC=0 and S=1) in any of the IA32_MCi banks of the processors, this is an 
unexpected condition for the MCE handler and the handler should signal the operating system to reset the 
system. 

° Before returning from the machine-check exception handler, software must clear the MCIP flag in the 
IA32_MCG_STATUS register. The MCIP flag is used to detect recursion. The machine-check architecture does 
not support recursion. When the processor receives a machine check when MCIP is set, it automatically enters 
the shutdown state. 


Example 15-4 gives pseudocode for an MC exception handler that supports recovery of UCR. 
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Example 15-4. Machine-Check Error Handler Pseudocode Supporting UCR 


MACHINE CHECK HANDLER: (* Called from INT 18 handler *) 
NOERROR = TRUE; 
ProcessorCount = 0; 
IF CPU supports MCA 
THEN 
RESTARTABILITY = TRUE; 
IF (Processor Family = 6 AND DisplayModel > OE€H) OR (Processor Family > 6) 
THEN 
IF (MCG_LMCE = 1) 
MCA_BROADCAST = FALSE; 
ELSE 
MCA_BROADCAST = TRUE; 
Fl; 
Acquire SpinLock; 
ProcessorCount++; (* Allowing one logical processor at a time to examine machine check registers *) 
CALL MCA ERROR PROCESSING; (* returns RESTARTABILITY and NOERROR *) 
ELSE 
MCA_BROADCAST = FALSE; 
(* Implement a rendezvous mechanism with the other processors if necessary *) 
CALL MCA ERROR PROCESSING; 
Fl; 
ELSE (* Pentium(R) processor compatible *) 
READ P5_MC_ADDR 
READ P5_MC_TYPE; 
RESTARTABILITY = FALSE; 
FI; 


IF NOERROR = TRUE 
THEN 
IF NOT (MCG_RIPV = 1 AND MCG_EIPV = 0) 
THEN 
RESTARTABILITY = FALSE; 
Fl 
Fl; 


IF RESTARTABILITY = FALSE 
THEN 
Report RESTARTABILITY to console; 
Reset system; 
Fl; 


IF MCA_BROADCAST = TRUE 
THEN 
IF ProcessorCount = MAX_PROCESSORS 
AND NOERROR = TRUE 
THEN 
Report RESTARTABILITY to console; 
Reset system; 
FI; 
Release SpinLock; 
Wait till Processorcount = MAX_PROCESSRS on system; 
(* implement a timeout and abort function if necessary *) 
Fl; 
CLEAR IA32_MCG_STATUS; 
RESUME Execution; 
(* End of MACHINE CHECK HANDLER*) 


MCA ERROR PROCESSING: (* MCA Error Processing Routine called from MCA Handler *) 
IF MCIP flag in |A32_MCG_STATUS = 0 
THEN (* MCIP=0 upon MCA is unexpected *) 
RESTARTABILITY = FALSE; 
Fl; 
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FOR each bank of machine-check registers 
DO 
CLEAR_MC_BANK = FALSE; 
READ IA32_MCi_STATUS; 
IF VAL Flag in |A32_MCi_STATUS = 1 
THEN 
IF UC Flag in |A32_MCi_STATUS = 1 
THEN 
IF Bit 24 in IA32_MCG_CAP = 0 
THEN (* the processor does not support software error recovery *) 
RESTARTABILITY = FALSE; 
NOERROR = FALSE; 
GOTO LOG MCA REGISTER; 
Fi; 
(* the processor supports software error recovery *) 
IF EN Flag in |A32_MCi_STATUS = 0 AND OVER Flag in |A32_MCi_STATUS=0 
THEN (* It is a spurious MCA Log. Log and clear the register *) 
CLEAR_MC_BANK = TRUE; 
GOTO LOG MCA REGISTER; 
Fi; 
IF PCC = 1 and EN = 1 in IA32_MCi_STATUS 
THEN (* processor context might have been corrupted *) 
RESTARTABILITY = FALSE; 
ELSE (* It is a uncorrected recoverable (UCR) error *) 
IF S Flag in |A32_MCi_STATUS = 0 
THEN 
IF AR Flag in |A32_MCi_STATUS = 0 
THEN (* It is a uncorrected no action required (UCNA) error *) 
GOTO CONTINUE; (* let CMCI and CMC polling handler to process *) 
ELSE 
RESTARTABILITY = FALSE; (* S=0, AR=1 is illegal *) 
Fl 
Fi; 
IF RESTARTABILITY = FALSE 
THEN (* no need to take recovery action if RESTARTABILITY is already false *) 
NOERROR = FALSE; 
GOTO LOG MCA REGISTER; 
Fi; 
(* Sin |A32_MCi_STATUS = 1 *) 
IF AR Flag in |A32_MCi_STATUS = 1 
THEN (* It is a software recoverable and action required (SRAR) error *) 
IF OVER Flag in IA32_MCi_STATUS = 1 
THEN 
RESTARTABILITY = FALSE; 
NOERROR = FALSE; 
GOTO LOG MCA REGISTER; 
Fl 
IF MCACOD Value in IA32_MCi_STATUS is recognized 
AND Current Processor is an Affected Processor 
THEN 
Implement MCACOD specific recovery action; 
CLEAR_MC_BANK = TRUE; 
ELSE 
RESTARTABILITY = FALSE; 
Fi; 
ELSE (* It is a software recoverable and action optional (SRAO) error *) 
IF OVER Flag in IA32_MCi_STATUS = 0 AND 
MCACOD in IA32_MCi_STATUS is recognized 
THEN 
Implement MCACOD specific recovery action; 
Fi; 
CLEAR_MC_BANK = TRUE; 
Fl; AR 
Fl; PCC 
NOERROR = FALSE; 
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GOTO LOG MCA REGISTER; 
ELSE (* It is a corrected error; continue to the next [A32_MCi_STATUS *) 
GOTO CONTINUE; 
Fl; UC 
Fl; VAL 


LOG MCA REGISTER: 


SAVE IA32_MCi_STATUS; 
If MISCV in IA32_MCi_STATUS 
THEN 
SAVE IA32_MCi_MISC; 
Fl; 
IF ADDRV in |A32_MCi_STATUS 
THEN 
SAVE IA32_MCi_ADDR; 
Fl; 
IF CLEAR_MC_BANK = TRUE 
THEN 
SET all O to IA32_MCi_STATUS; 
If MISCV in IA32_MCi_STATUS 
THEN 
SET all 0 to IA32_MCi_MISC; 
Fl; 
IF ADDRV in IA32_MCi_STATUS 
THEN 
SET all 0 to IA32_MCi_ADDR; 
Fl; 
Fl; 
CONTINUE: 


OD; 


(*END FOR *) 
RETURN; 
(* End of MCA ERROR PROCESSING*) 


15.10.4.2 Corrected Machine-Check Handler for Error Recovery 


When writing a corrected machine check handler, which is invoked as a result of CMCI or called from an OS CMC 
Polling dispatcher, consider the following: 


The VAL (valid) flag in each IA32_MCi_STATUS register indicates whether the error information in the register 
is valid. If this flag is clear, the registers in that bank does not contain valid error information and does not need 
to be checked. 


The CMCI or CMC polling handler is responsible for logging and clearing corrected errors. The UC flag in each 
IA32_MCi_Status register indicates whether the reported error was corrected (UC=0) or not (UC=1). 


When IA32_MCG_CAP [24] is one, the CMC handler is also responsible for logging and clearing uncorrected no- 
action required (UCNA) errors. When the UC flag is one but the PCC, S, and AR flags are zero in the 
IA32_MCi_STATUS register, the reported error in this bank is an uncorrected no-action required (UCNA) error. 
In cases when SRAO error are signaled as UCNA error via CMCI, software can perform recovery for those errors 
identified in Table 15-16. 


In addition to corrected errors and UCNA errors, the CMC handler optionally logs uncorrected (UC=1 and 
PCC=1), software recoverable machine check errors (UC=1, PCC=0 and S=1), but should avoid clearing those 
errors from the MC banks. Clearing these errors may result in accidentally removing these errors before these 
errors are actually handled and processed by the MCE handler for attempted software error recovery. 


Example 15-5 gives pseudocode for a CMCI handler with UCR support. 


Vol. 3B 15-35 


MACHINE-CHECK ARCHITECTURE 


Example 15-5. Corrected Error Handler Pseudocode with UCR Support 


Corrected Error HANDLER: (* Called from CMCI handler or OS CMC Polling Dispatcher*) 
IF CPU supports MCA 
THEN 
FOR each bank of machine-check registers 
DO 
READ IA32_MCi_STATUS; 
IF VAL flag in |A32_MCi_STATUS = 1 


THEN 
IF UC Flag in |A32_MCi_STATUS = 0 (* It is a corrected error *) 
THEN 
GOTO LOG CMC ERROR; 
ELSE 
IF Bit 24 in IA32_MCG_CAP = 0 
THEN 
GOTO CONTINUE; 
Fi; 
IF S Flag in |A32_MCi_STATUS = 0 AND AR Flag in IA32_MCi_STATUS = 0 
THEN (* It is a uncorrected no action required error *) 
GOTO LOG CMC ERROR 
Fl 
IF EN Flag in |A32_MCi_STATUS = 0 
THEN (* It is a spurious MCA error *) 
GOTO LOG CMC ERROR 
Fi; 
Fi; 
Fi; 
GOTO CONTINUE; 
LOG CMC ERROR: 


SAVE IA32_MCi_STATUS; 
If MISCV Flag in IA32_MCi_STATUS 
THEN 
SAVE IA32_MCi_MISC; 
SET all 0 to IA32_MCi_MISC; 
Fl; 
IF ADDRV Flag in |A32_MCi_STATUS 
THEN 
SAVE IA32_MCi_ADDR; 
SET all 0 to |A32_MCi_ADDR 
Fl; 
SET all 0 to IA32_MCi_STATUS; 
CONTINUE: 
OD; 
( *END FOR *) 
Fl; 
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INTERPRETING MACHINE-CHECK 
ERROR CODES 


Encoding of the model-specific and other information fields is different across processor families. The differences 
are documented in the following sections. 


16.1. INCREMENTAL DECODING INFORMATION: PROCESSOR FAMILY O6H 
MACHINE ERROR CODES FOR MACHINE CHECK 


Section 16.1 provides information for interpreting additional model-specific fields for external bus errors relating to 
processor family O6H. The references to processor family 06H refers to only IA-32 processors with CPUID signa- 
tures listed in Table 16-1. 


Table 16-1. CPUID DisplayFamily_DisplayModel Signatures for Processor Family O6H 


DisplayFamily_DisplayModel Processor Families/Processor Number Series 

06_0EH Intel Core Duo, Intel Core Solo processors 

06_0DH Intel Pentium M processor 

06_09H Intel Pentium M processor 

06_7H, 06_08H, 06_0AH, 06_0BH Intel Pentium III] Xeon Processor, Intel Pentium III Processor 
06_03H, 06_05H Intel Pentium II Xeon Processor, Intel Pentium I] Processor 
06_01H Intel Pentium Pro Processor 


These errors are reported in the IA32_MCi_STATUS MSRs. They are reported architecturally as compound errors 
with a general form of 0000 1PPT RRRR IILL in the MCA error code field. See Chapter 15 for information on the 
interpretation of compound error codes. Incremental decoding information is listed in Table 16-2. 


Table 16-2. Incremental Decoding Information: Processor Family O6H Machine Error Codes For Machine Check 


Type Bit No. | Bit Function Bit Description 
MCA error 15:0 

codes! 

Model specific | 18:16 | Reserved Reserved 


errors 
Model specific | 24:19 | Bus queue request 000000 for BQ_DCU_READ_TYPE error 
errors type 000010 for BQ_IFU_DEMAND_TYPE error 
000011 for BQ_IFU_DEMAND_NC_TYPE error 
000100 for BQ_DCU_RFO_TYPE error 
000101 for BQ_DCU_RFO_LOCK_TYPE error 


000110 for BQ_DCU_ITOM_TYPE error 
001000 for BOQ_DCU_WB_TYPE error 
001010 for BQ_DCU_WCEVICT_TYPE error 
001011 for BQ_DCU_WCLINE_TYPE error 
001100 for BQ_DCU_BTM_TYPE error 
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Table 16-2. Incremental Decoding Information: Processor Family O6H Machine Error Codes For Machine Check 


Type 


Bit No. 


Bit Function 


Bit Description 


001101 for BQ_DCU_INTACK_TYPE error 
001110 for BQ_DCU_INVALL2_TYPE error 
001111 for BQ_DCU_FLUSHL2_TYPE error 
010000 for BQ_DCU_PART_RD_TYPE error 
010010 for BQ_.DCU_PART_WR_TYPE error 
010100 for BQ_DCU_SPEC_CYC_TYPE error 
011000 for BQ_DCU_IO_RD_TYPE error 
011001 for BQ_DCU_IO_WR_TYPE error 
011100 for BQ_DCU_LOCK_RD_TYPE error 
011110 for BQ_DCU_SPLOCK_RD_TYPE error 
011101 for BQ_DCU_LOCK_WR_TYPE error 


Model specific | 27:25 


errors 


Bus queue error type 


000 for BQ_ERR_HARD_TYPE error 
001 for BQ_ERR_DOUBLE_TYPE error 
010 for BQ_ERR_AERR2_TYPE error 
100 for BQ_ERR_SINGLE_TYPE error 
101 for BQ_ERR_AERR1_TYPE error 


Model specific | 28 FRC error 1 if FRC error active 
errors 

29 BERR 1 if BERR is driven 

30 Internal BINIT 1 if BINIT driven for this processor 

31 Reserved Reserved 

Other 34:32 | Reserved Reserved 
information 

35 External BINIT 1 if BINIT is received from external bus. 

36 Response parity error | This bit is asserted in |A32_MCi_STATUS if this component has received a parity 
error on the RS[2:0]# pins for a response transaction. The RS signals are checked 
by the RSP# external pin. 

37 Bus BINIT This bit is asserted in IA32_MCi_STATUS if this component has received a hard 
error response on a split transaction one access that has needed to be split across 
the 64-bit external bus interface into two accesses). 

38 Timeout BINIT This bit is asserted in IA32_MCi_STATUS if this component has experienced a ROB 
time-out, which indicates that no micro-instruction has been retired for a 
predetermined period of time. 

A ROB time-out occurs when the 15-bit ROB time-out counter carries a 1 out of its 
high order bit. 2 The timer is cleared when a micro-instruction retires, an exception 
is detected by the core processor, RESET is asserted, or when a ROB BINIT occurs. 
The ROB time-out counter is prescaled by the 8-bit PIC timer which is a divide by 
128 of the bus clock the bus clock is 1:2, 1:3, 1:4 of the core clock). When a carry 
out of the 8-bit PIC timer occurs, the ROB counter counts up by one. While this bit 
is asserted, it cannot be overwritten by another error. 

41:39 | Reserved Reserved 

42 Hard error This bit is asserted in IA32_MCi_STATUS if this component has initiated a bus 


transactions which has received a hard error response. While this bit is asserted, it 
cannot be overwritten. 
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Table 16-2. Incremental Decoding Information: Processor Family O6H Machine Error Codes For Machine Check 
Type BitNo. | Bit Function Bit Description 


43 IERR This bit is asserted in [A32_MCi_STATUS if this component has experienced a 
failure that causes the IERR pin to be asserted. While this bit is asserted, it cannot 
be overwritten. 


44 AERR This bit is asserted in [A32_MCi_STATUS if this component has initiated 2 failing 
bus transactions which have failed due to Address Parity Errors AERR asserted). 
While this bit is asserted, it cannot be overwritten. 


45 UECC The Uncorrectable ECC error bit is asserted in IA32_MCi_STATUS for uncorrected 
ECC errors. While this bit is asserted, the ECC syndrome field will not be 
overwritten. 

46 CECC The correctable ECC error bit is asserted in |A32_MCi_STATUS for corrected ECC 
errors. 

54:47 | ECC syndrome The ECC syndrome field in IA32_MCi_STATUS contains the 8-bit ECC syndrome only 


if the error was a correctable/uncorrectable ECC error and there wasn't a previous 
valid ECC error syndrome logged in |A32_MCi_STATUS. 

A previous valid ECC error in |A32_MCi_STATUS is indicated by 
IA32_MCi_STATUS.bit45 uncorrectable error occurred) being asserted. After 
processing an ECC error, machine-check handling software should clear 
1A32_MCi_STATUS.bit45 so that future ECC error syndromes can be logged. 


56:55 | Reserved Reserved. 
Status register | 63:57 
validity 
indicators? 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
2. For processors with a CPUID signature of O6_OEH, a ROB time-out occurs when the 23-bit ROB time-out counter carries a 1 out of its 
high order bit. 


16.2 INCREMENTAL DECODING INFORMATION: INTEL CORE 2 PROCESSOR 
FAMILY MACHINE ERROR CODES FOR MACHINE CHECK 


Table 16-4 provides information for interpreting additional model-specific fields for external bus errors relating to 
processor based on Intel Core microarchitecture, which implements the P4 bus specification. Table 16-3 lists the 
CPUID signatures for Intel 64 processors that are covered by Table 16-4. These errors are reported in the 
IA32_MCi_STATUS MSRs. They are reported architecturally as compound errors with a general form of 

0000 1PPT RRRR IILL in the MCA error code field. See Chapter 15 for information on the interpretation of 
compound error codes. 


Table 16-3. CPUID DisplayFamily_DisplayModel Signatures for Processors Based on Intel Core Microarchitecture 
DisplayFamily_DisplayModel | Processor Families/Processor Number Series 


06_1DH Intel Xeon Processor 7400 series. 
06_17H Intel Xeon Processor 5200, 5400 series, Intel Core 2 Quad processor Q9650. 
06_OFH Intel Xeon Processor 3000, 3200, 5100, 5300, 7300 series, Intel Core 2 Quad, Intel Core 2 Extreme, 


Intel Core 2 Duo processors, Intel Pentium dual-core processors. 
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Table 16-4. Incremental Bus Error Codes of Machine Check for Processors 
Based on Intel Core Microarchitecture 


Type Bit No. | Bit Function Bit Description 
MCA error 15:0 

codes! 

Model specific | 18:16 | Reserved Reserved 


errors 


Model specific | 24:19 | Bus queue request ‘000001 for BQ_PREF_READ_TYPE error 
errors type 000000 for BOQ_DCU_READ_TYPE error 
000010 for BQ_IFU_DEMAND_TYPE error 
000011 for BQ_IFU_LDEMAND_NC_TYPE error 
000100 for BQ_DCU_RFO_TYPE error 
000101 for BQ_DCU_RFO_LOCK_TYPE error 


000110 for BQ_DCU_ITOM_TYPE error 
001000 for BQ_DCU_WB_TYPE error 
001010 for BQ_DCU_WCEVICT_TYPE error 
001011 for BQ_DCU_WCLINE_TYPE error 
001100 for BQ_DCU_BTM_TYPE error 


001101 for BQ_DCU_INTACK_TYPE error 
001110 for BQ_DCU_INVALL2_TYPE error 
001111 for BQ_DCU_FLUSHL2_TYPE error 
010000 for BQ_DCU_PART_RD_TYPE error 
010010 for BQ_DCU_PART_WR_TYPE error 


010100 for BQ_DCU_SPEC_CYC_TYPE error 
011000 for BQ_DCU_IO_RD_TYPE error 
011001 for BQ_DCU_IO_WR_TYPE error 
011100 for BQ_DCU_LOCK_RD_TYPE error 
011110 for BQ_.DCU_SPLOCK_RD_TYPE error 
011101 for BQ_DCU_LOCK_WR_TYPE error 
100100 for BQ_L2_WI_RFO_TYPE error 
100110 for BQ_L2_WI_ITOM_TYPE error 


Model specific | 27:25 | Bus queue error type | ‘001 for Address Parity Error 


errors ‘010 for Response Hard Error 
‘011 for Response Parity Error 

Model specific | 28 MCE Driven 1 if MCE is driven 
errors 

29 MCE Observed 1 if MCE is observed 

30 Internal BINIT 1 if BINIT driven for this processor 

31 BINIT Observed 1 if BINIT is observed for this processor 
Other 33:32 | Reserved Reserved 
information 

34 PIC and FSB data Data Parity detected on either PIC or FSB access 

parity 
35 Reserved Reserved 
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Table 16-4. Incremental Bus Error Codes of Machine Check for Processors 
Based on Intel Core Microarchitecture (Contd.) 


validity 
indicators 


NOTES: 


1 


Type BitNo. | Bit Function Bit Description 

36 Response parity error | This bit is asserted in |A32_MCi_STATUS if this component has received a parity 
error on the RS[2:0]# pins for a response transaction. The RS signals are checked 
by the RSP# external pin. 

37 FSB address parity Address parity error detected: 

1 = Address parity error detected 
O = No address parity error 

38 Timeout BINIT This bit is asserted in [A32_MCi_STATUS if this component has experienced a ROB 
time-out, which indicates that no micro-instruction has been retired for a 
predetermined period of time. 

A ROB time-out occurs when the 23-bit ROB time-out counter carries a 1 out of its 
high order bit. The timer is cleared when a micro-instruction retires, an exception is 
detected by the core processor, RESET is asserted, or when a ROB BINIT occurs. 
The ROB time-out counter is prescaled by the 8-bit PIC timer which is a divide by 
128 of the bus clock the bus clock is 1:2, 1:3, 1:4 of the core clock). When a carry 
out of the 8-bit PIC timer occurs, the ROB counter counts up by one. While this bit 
is asserted, it cannot be overwritten by another error. 

41:39 | Reserved Reserved 

42 Hard error This bit is asserted in [A32_MCi_STATUS if this component has initiated a bus 
transactions which has received a hard error response. While this bit is asserted, it 
cannot be overwritten. 

43 IERR This bit is asserted in IA32_MCi_STATUS if this component has experienced a 
failure that causes the IERR pin to be asserted. While this bit is asserted, it cannot 
be overwritten. 

44 Reserved Reserved 

45 Reserved Reserved 

46 Reserved Reserved 

54:47 | Reserved Reserved 

56:55 | Reserved Reserved. 

Status register | 63:57 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.2.1 


section. 


Model-Specific Machine Check Error Codes for Intel Xeon Processor 7400 Series 


Intel Xeon processor 7400 series has machine check register banks that generally follows the description of 
Chapter 15 and Section 16.2. Additional error codes specific to Intel Xeon processor 7400 series is describe in this 


MC4_STATUS[63:0] is the main error logging for the processor’s L3 and front side bus errors for Intel Xeon 
processor 7400 series. It supports the L3 Errors, Bus and Interconnect Errors Compound Error Codes in the MCA 
Error Code Field. 
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16.2.1.1 Processor Machine Check Status Register 
Incremental MCA Error Code Definition 


Intel Xeon processor 7400 series use compound MCA Error Codes for logging its Bus internal machine check 
errors, L3 Errors, and Bus/Interconnect Errors. It defines incremental Machine Check error types 
(IA32_MC6_STATUS[15:0]) beyond those defined in Chapter 15. Table 16-5 lists these incremental MCA error 
code types that apply to IA32_MC6_STATUS. Error code details are specified in MC6_STATUS [31:16] (see 
Section 16.2.2), the “Model Specific Error Code” field. The information in the “Other_Info” field 
(MC4_STATUS[56:32]) is common to the three processor error types and contains a correctable event count and 
specifies the MC6_MISC register format. 


Table 16-5. Incremental MCA Error Code Types for Intel Xeon Processor 7400 


Processor MCA_Error_Code (MC6_STATUS[15:0]) 


Type | Error Code Binary Encoding Meaning 

C Internal Error | 0000 0100 0000 0000 | Internal Error Type Code 

B Bus and 0000 100x 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 
Interconnect implementations 
Error 0000 101x 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 


implementations 


0000 110x 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 
implementations 


0000 1110 0000 1111 | Bus and Interconnection Error Type Code 


0000 1111 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 
implementations 


The Bold faced binary encodings are the only encodings used by the processor for MC4_STATUS[15:0]. 


16.2.2 Intel Xeon Processor 7400 Model Specific Error Code Field 


16.2.2.1 Processor Model Specific Error Code Field 
Type B: Bus and Interconnect Error 


Note: The Model Specific Error Code field in MC6_STATUS (bits 31:16). 


Table 16-6. Type B Bus and Interconnect Error Codes 


BitNum_ | Sub-Field Name Description 

16 FSB Request Parity Parity error detected during FSB request phase 

19:17 Reserved 

20 FSB Hard Fail Response “Hard Failure” response received for a local transaction 
21 FSB Response Parity Parity error on FSB response field detected 

22 FSB Data Parity FSB data parity error on inbound data detected 

31:23 “- Reserved 
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Type C: Cache Bus Controller Error 


INTERPRETING MACHINE-CHECK ERROR CODES 


Table 16-7. Type C Cache Bus Controller Error Codes 


MC4_STATUS[31:16] (MSCE) Value 


Error Description 


0000_0000_0000_0001 0001H 


Inclusion Error from Core 0 


0000_0000_0000_0010 0002H 


Inclusion Error from Core 1 


0000_0000_0000_0011 0003H 


Write Exclusive Error from Core 0 


0000_0000_0000_0100 0004H 


Write Exclusive Error from Core 1 


0000_0000_0000_0101 O005H 


Inclusion Error from FSB 


0000_0000_0000_0110 OO06H 


SNP Stall Error from FSB 


0000_0000_0000_0111 0007H 


Write Stall Error from FSB 


0000_0000_0000_1000 0008H 
0000_0000_0000_1010 OOOAH 


FSB Arb Timeout Error 


Inclusion Error from Core 2 


0000_0000_0000_1011 OOOBH 


Write Exclusive Error from Core 2 


0000_0010_0000_0000 0200H 
0000_0011_0000_0000 0300H 


Internal Timeout error 


Internal Timeout Error 


0000_0100_0000_0000 0400H 


Intel? Cache Safe Technology Queue Full Error or Disabled-ways-in-a-set overflow 


0000_0101_0000_0000 0500H 
1100_0000_0000_0010 C002H 


Quiet cycle Timeout Error (correctable) 


Correctable ECC event on outgoing Core O data 


1100_0000_0000_0100 CO04H 


Correctable ECC event on outgoing Core 1 data 


1100_0000_0000_1000 CO08H 
1110_0000_0000_0010 E002H 


Correctable ECC event on outgoing Core 2 data 


Uncorrectable ECC error on outgoing Core O data 


1110_0000_0000_0100 E004H 


Uncorrectable ECC error on outgoing Core 1 data 


1110_0000_0000_1000 E008H 


— all other encodings — 


Uncorrectable ECC error on outgoing Core 2 data 


Reserved 


16.3. INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR 
3400, 3500, 5500 SERIES, MACHINE ERROR CODES FOR MACHINE CHECK 


Table 16-8 through Table 16-12 provide information for interpreting additional model-specific fields for memory 
controller errors relating to the Intel® Xeon® processor 3400, 3500, 5500 series with CPUID 
DisplayFamily_DisplaySignature 06_1AH, which supports Intel QuickPath Interconnect links. Incremental MC error 
codes related to the Intel QPI links are reported in the register banks IA32_MCO and IA32_MC1, incremental error 
codes for internal machine check is reported in the register bank IA32_MC7, and incremental error codes for the 


memory controller unit is reported in the register banks IA32_MC8. 
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16.3.1 Intel QPI Machine Check Errors 


Table 16-8. Intel QPI Machine Check Error Codes for |A32_MCO_STATUS and IA32_MC1_STATUS 


Type Bit No. | Bit Function Bit Description 

MCA error codes! 15:0 MCACOD Bus error format: 1PPTRRRRIILL 

Model specific errors 
16 Header Parity if 1, QP! Header had bad parity 
17 Data Parity If 1, QP] Data packet had bad parity 
18 Retries Exceeded If 1, number of QPI retries was exceeded 
19 Received Poison if 1, Received a data packet that was marked as poisoned by the sender 
21:20 | Reserved Reserved 
22 Unsupported Message If 1, QPI received a message encoding it does not support 
23 Unsupported Credit If 1, QPI credit type is not supported. 
24 Receive Flit Overrun If 1, Sender sent too many QPI flits to the receiver. 
25 Received Failed If 1, Indicates that sender sent a failed response to receiver. 

Response 

26 Receiver Clock Jitter If 1, clock jitter detected in the internal QPI clocking 
56:27 | Reserved Reserved 

Status register 63:57 

validity indicators? 

NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


Table 16-9. Intel QPI Machine Check Error Codes for [A32_MCO_MISC and IA32_MC1_MISC 


Type BitNo. | Bit Function Bit Description 
Model specific errors! 
7:0 QPI Opcode Message class and opcode from the packet with the error 
13:8 RTld QP! Request Transaction ID 
15:14 | Reserved Reserved 
18:16 | RHNID QP! Requestor/Home Node ID 
23:19 | Reserved Reserved 
24 IIB QPI Interleave/Head Indication Bit 


NOTES: 
1. Which of these fields are valid depends on the error type. 


16.3.2 Internal Machine Check Errors 


Table 16-10. Machine Check Error Codes for [A32_MC7_STATUS 


Type Bit No. | Bit Function Bit Description 


MCA error codes! 15:0 MCACOD 


Model specific errors 
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Type Bit No. | Bit Function Bit Description 
23:16 | Reserved Reserved 
31:24 | Reserved except for | OOh - No Error 
the following O3h - Reset firmware did not complete 
O8h - Received an invalid CMPD 
OAh - Invalid Power Management Request 
ODh - Invalid S-state transition 
11h - VID controller does not match POC controller selected 
1Ah - MSID from POC does not match CPU MSID 
56:32 | Reserved Reserved 
Status register validity | 63:57 
indicators? 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.3.3. Memory Controller Errors 
Table 16-11. Incremental Memory Controller Error Codes of Machine Check for IA32_MC8_STATUS 
Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Memory error format: 1MMMCCCC 
Model specific errors 
16 Read ECC error if 1, ECC occurred on a read 
17 RAS ECC error If 1, ECC occurred on a scrub 
18 Write parity error If 1, bad parity on a write 
19 Redundancy loss if 1, Error in half of redundant memory 
20 Reserved Reserved 
21 Memory range error | If 1, Memory access out of range 
22 RTID out of range If 1, Internal ID invalid 
23 Address parity error | If 1, bad address parity 
24 Byte enable parity If 1, bad enable parity 
error 
Other information 37:25 | Reserved Reserved 
52:38 | CORE_ERR_CNT Corrected error count 
56:53 | Reserved Reserved 
Status register validity | 63:57 
indicators1 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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Table 16-12. Incremental Memory Controller Error Codes of Machine Check for |A32_MC8_MISC 


Type BitNo. | Bit Function Bit Description 

Model specific errors! 
7:0 RTld Transaction Tracker ID 
15:8 Reserved Reserved 
17:16 | DIMM DIMM ID which got the error 
19:18 | Channel Channel ID which got the error 
31:20 | Reserved Reserved 
63:32 | Syndrome ECC Syndrome 


NOTES: 
1. Which of these fields are valid depends on the error type. 


16.4 INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR E5 
FAMILY, MACHINE ERROR CODES FOR MACHINE CHECK 


Table 16-13 through Table 16-15 provide information for interpreting additional model-specific fields for memory 
controller errors relating to the Intel® Xeon® processor E5 Family with CPUID DisplayFamily_DisplaySignature 
06_2DH, which supports Intel QuickPath Interconnect links. Incremental MC error codes related to the Intel QPI 
links are reported in the register banks IA32_MC6 and IA32_MC7, incremental error codes for internal machine 
check error from PCU controller is reported in the register bank IA32_MC4, and incremental error codes for the 
memory controller unit is reported in the register banks IA32_MC8-IA32_MCi1. 


16.4.1 Internal Machine Check Errors 


Table 16-13. Machine Check Error Codes for |A32_MC4_STATUS 


Type BitNo. | Bit Function Bit Description 
MCA error 15:0 MCACOD 
codes! 


Model specific | 19:16 | Reserved except for | OOOOb - No Error 

errors the following 0001b - Non_IMem_Sel 
0010b - |_Parity_Error 
0011b - Bad_OpCode 
0100b - |_Stack_Underflow 
0101b - |_Stack_Overflow 
0110b - D_Stack_Underflow 
0111b - D_Stack_Overflow 
1000b - Non-DMem_Sel 
1001b - D_Parity_€rror 
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Type 


Bit No. 


Bit Function 


Bit Description 


Status register 
validity 


indicators? 


23:20 


Reserved 


Reserved 


31:24 


Reserved except for 
the following 


OOh - No Error 

ODh - MC_IMC_FORCE_SR_S3_ TIMEOUT 

OEh - MC_CPD_UNCPD_ST_TIMEOUT 

OFh - MC_PKGS_SAFE_WP_TIMEOUT 

43h - MC_PECI_MAILBOX_QUIESCE_TIMEOUT 
5Ch - MC_MORE_THAN_ONE_LT_AGENT 

60h - MC_INVALID_PKGS_REQ_PCH 

61h - MC_INVALID_PKGS_REQ_QPI 

62h - MC_INVALID_PKGS_RES_QPI 

63h - MC_INVALID_PKGC_RES_PCH 

64h - MC_INVALID_PKG_STATE_CONFIG 

70h - MC_WATCHDG_TIMEOUT_PKGC_SECONDARY 
71h - MC_WATCHDG_TIMEOQUT_PKGC_MAIN 

72h - MC_WATCHDG_TIMEOQUT_PKGS_MAIN 
7ah - MC_HA_FAILSTS_CHANGE_DETECTED 

81h - MC_RECOVERABLE_DIE_THERMAL_TOO_HOT 


56:32 
63:57 


Reserved 


Reserved 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.4.2 Intel QPI Machine Check Errors 

Table 16-14. Intel QPI MC Error Codes for |A32_MC6_STATUS and IA32_MC7_STATUS 
Type BitNo. | Bit Function Bit Description 
MCA error 15:0 MCACOD Bus error format: 1PPTRRRRIILL 
codes! 
Model specific 
errors 

56:16 | Reserved Reserved 

Status register | 63:57 
validity 
indicators? 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.4.3 


Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC8_STATUS- 
IA32_MC1ii_STATUS. The supported error codes are follows the architectural MCACOD definition type 1MMMCCCC 
(see Chapter 15, “Machine-Check Architecture,”). MSR_ERROR_CONTROL.[bit 1] can enable additional informa- 
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tion logging of the IMC. The additional error information logged by the IMC is stored in IA32_MCi_STATUS and 
IA32_MCi_MISC; (i = 8, 11). 


Table 16-15. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 8, 11) 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Bus error format: 1PPTRRRRIILL 
Model specific 31:16 | Reserved except for | 001H - Address parity error 
errors the following 002H - HA Wrt buffer Data parity error 
O004H - HA Wrt byte enable parity error 
OO8H - Corrected patrol scrub error 
010H - Uncorrected patrol scrub error 
O20H - Corrected spare error 
O40H - Uncorrected spare error 
Model specific 36:32 | Other info When MSR_ERROR_CONTROL[1] is set, allows the iMC to log first device 
errors error when corrected error is detected during normal read. 
37 Reserved Reserved 
56:38 See Chapter 15, “Machine-Check Architecture,” 
Status register 63:57 
validity indicators? 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


Table 16-16. Intel IMC MC Error Codes for IA32_MCi_MISC (i= 8, 11) 


Type BitNo. | Bit Function Bit Description 
MCA addr info! | 8:0 See Chapter 15, “Machine-Check Architecture,” 
Model specific | 13:9 * When MSR_ERROR_CONTROL|1] is set, allows the iMC to log second device 
errors error when corrected error is detected during normal read. 
* Otherwise contain parity error if MCi_Status indicates HA_WB_Data or 
HA_W_BE parity error. 
Model specific | 29:14 | ErrMask_1stErrDev When MSR_ERROR_CONTROL|1] is set, allows the iMC to log first-device error bit 
errors mask. 
Model specific | 45:30 | ErrMask_2nd€rrDev | When MSR_ERROR_CONTROL|[1] is set, allows the iMC to log second-device error 
errors bit mask. 
50:46 | FailRank_1st€rrDev | When MSR_ERROR_CONTROL|1] is set, allows the iMC to log first-device error 
failing rank. 
55:51. | FailRank_2ndE€rrDev | When MSR_ERROR_CONTROL|1] is set, allows the iMC to log second-device error 
failing rank. 
58:56 | Reserved Reserved 
61:59 | Reserved Reserved 
62 Valid_1st€rrDev When MSR_ERROR_CONTROL[1] is set, indicates the iMC has logged valid data 
from the first correctable error in a memory device. 
63 Valid_2ndErrDev When MSR_ERROR_CONTROL,[1] is set, indicates the iMC has logged valid data due 


to a second correctable error in a memory device. Use this information only after 
there is valid first error info indicated by bit 62. 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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16.5 INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR E5 
V2 AND INTEL® XEON® PROCESSOR E7 V2 FAMILIES, MACHINE ERROR 
CODES FOR MACHINE CHECK 


The Intel® Xeon® processor E5 v2 family and the Intel® Xeon® processor E7 v2 family are based on the Ivy 
Bridge-EP microarchitecture and can be identified with CPUID DisplayFamily_DisplaySignature 06_3EH. Incre- 
mental error codes for internal machine check error from PCU controller is reported in the register bank IA32_MC4, 
Table lists model-specific fields to interpret error codes applicable to IA32_MC4_STATUS. Incremental MC error 
codes related to the Intel QPI links are reported in the register banks IA32_MCS5. Information listed in Table 16-14 
for QPI MC error code apply to IA32_MC5_STATUS. Incremental error codes for the memory controller unit is 
reported in the register banks IA32_MC9-IA32_MC16. Table 16-18 lists model-specific error codes apply to 
IA32_MCi_STATUS, i = 9-16. 


16.5.1 Internal Machine Check Errors 


Table 16-17. Machine Check Error Codes for [A32_MC4_STATUS 
Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD 


Model specific errors | 19:16 | Reserved except for | OOOOb - No Error 

the following 0001b - Non_IMem_Sel 
0010b - |_Parity_Error 
0011b - Bad_OpCode 
0100b - |_Stack_Underflow 
0101b - |_Stack_Overflow 
0110b - D_Stack_Underflow 
0111b - D_Stack_Overflow 
1000b - Non-DMem_Sel 
1001b - D_Parity_Error 


23:20 | Reserved Reserved 
31:24 | Reserved except for | 00h - No Error 
the following ODh - MC_IMC_FORCE_SR_S3_TIMEOUT 


OEh - MC_CPD_UNCPD_ST_TIMEOUT 

OFh - MC_PKGS_SAFE_WP_TIMEOQUT 

43h - MC_PECI_MAILBOX_QUIESCE_TIMEOQUT 
44h - MC_CRITICAL_VR_FAILED 

45h - MC_ICC_MAX-NOTSUPPORTED 

5Ch - MC_MORE_THAN_ONE_LT_AGENT 

60h - MC_INVALID_PKGS_REQ_PCH 

61h - MC_INVALID_PKGS_REQ_QPI 

62h - MC_INVALID_PKGS_RES_QPI 

63h - MC_INVALID_PKGC_RES_PCH 

64h - MC_INVALID_PKG_STATE_CONFIG 

70h - MC_WATCHDG_TIMEOUT_PKGC_SECONDARY 
71h - MC_WATCHDG_TIMEOUT_PKGC_MAIN 
72h - MC_WATCHDG_TIMEOUT_PKGS_MAIN 
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Type BitNo. | Bit Function Bit Description 
7Ah - MC_HA_FAILSTS_CHANGE_DETECTED 
7Bh - MC_PCIE_R2PCIE-RW_BLOCK_ACK_TIMEOUT 
81h - MC_RECOVERABLE_DIE_THERMAL_TOO_HOT 
56:32 | Reserved Reserved 
Status register 63:57 
validity indicators? 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.5.2 


Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC9_STATUS- 
IA32_MC16_STATUS. The supported error codes are follows the architectural MCACOD definition type 1MMMCCCC 
(see Chapter 15, “Machine-Check Architecture”). 


MSR_ERROR_CONTROL. [bit 1] can enable additional information logging of the IMC. The additional error informa- 
tion logged by the IMC is stored in IA32_MCi_STATUS and IA32_MCi_MISC; (i = 9-16). 


IA32_MCi_STATUS (i=9-12) log errors from the first memory controller. The second memory controller logs into 
IA32_MCi_STATUS (i=13-16). 


Table 16-18. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 9-16) 


Type Bit No. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Memory Controller error format: OOOF OO00 1MMM CCCC 
Model specific 31:16 | Reserved except for | 001H - Address parity error 
errors the following 002H - HA Wrt buffer Data parity error 
O04H - HA Wrt byte enable parity error 
OOS8H - Corrected patrol scrub error 
010H - Uncorrected patrol scrub error 
O20H - Corrected spare error 
O40H - Uncorrected spare error 
O80H - Corrected memory read error. (Only applicable with iMC’s “Additional 
Error logging” Mode-1 enabled.) 
100H - iMC, WDB, parity errors 
36:32 | Other info When MSR_ERROR_CONTROL[1] is set, logs an encoded value from the first error 
device. 
37 Reserved Reserved 
56:38 See Chapter 15, “Machine-Check Architecture,” 
Status register 63:57 
validity indicators? 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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Table 16-19. Intel IMC MC Error Codes for IA32_MCi_MISC (i= 9-16) 


Type BitNo. | Bit Function Bit Description 

MCA addr info! | 8:0 See Chapter 15, “Machine-Check Architecture,” 

Model specific | 13:9 If the error logged is MCWrDataPar error or MCWrBEPar error, this field is the WDB 
errors ID that has the parity error. OR if the second error logged is a correctable read 


error, MC logs the second error device in this field. 


Model specific | 29:14 | ErrMask_1st€rrDev When MSR_ERROR_CONTROL|1] is set, allows the iMC to log first-device error bit 
errors mask. 


Model specific | 45:30 | ErrMask_2ndErrDev | When MSR_ERROR_CONTROL|1] is set, allows the iMC to log second-device error 
errors bit mask. 


50:46 | FailRank_1st€rrDev | When MSR_ERROR_CONTROL,1] is set, allows the iMC to log first-device error 


failing rank. 

55:51 | FailRank_2ndErrDev | When MSR_ERROR_CONTROL[1] is set, allows the iMC to log second-device error 
failing rank. 

61:56 Reserved 

62 Valid_1st€rrDev When MSR_ERROR_CONTROL[1] is set, indicates the iMC has logged valid data 
from a correctable error from memory read associated with first error device. 

63 Valid_2ndErrDev When MSR_ERROR_CONTROL,[1] is set, indicates the iIMC has logged valid data due 


to a second correctable error in a memory device. Use this information only after 
there is valid first error info indicated by bit 62. 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.5.3. Home Agent Machine Check Errors 


Memory errors from the first memory controller may be logged in the IA32_MC7_{STATUS,ADDR, MISC} registers, 
while the second memory controller logs to IA32_MC8_{STATUS,ADDR,MISC}. 


16.6 INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR E5 
V3 FAMILY, MACHINE ERROR CODES FOR MACHINE CHECK 


The Intel® Xeon® processor E5 v3 family is based on the Haswell-E microarchitecture and can be identified with 
CPUID DisplayFamily_DisplaySignature 06_3FH. Incremental error codes for internal machine check error from 
PCU controller is reported in the register bank IA32_MC4, Table 16-20 lists model-specific fields to interpret error 
codes applicable to IA32_MC4_STATUS. Incremental MC error codes related to the Intel QPI links are reported in 
the register banks IA32_MC5, IA32_MC20, and IA32_MC21. Information listed in Table 16-21 for QPI MC error 
codes. Incremental error codes for the memory controller unit is reported in the register banks IA32_MC9- 
IA32_MC16. Table 16-22 lists model-specific error codes apply to IA32_MCi_STATUS, i = 9-16. 
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16.6.1 Internal Machine Check Errors 


Table 16-20. Machine Check Error Codes for [A32_MC4_STATUS 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD 
MCACOD2 15:0 Internal Errors 0402h - PCU internal Errors 


0403h - PCU internal Errors 
0406h - Intel TXT Errors 
0407h - Other UBOX internal Errors. 


On an IERR caused by a core 3-strike the [A32_MC3_STATUS (MLC) is copied 
to the [A32_MC4_STATUS (After a 3-strike, the core MCA banks will be 


unavailable). 
Model specific errors | 19:16 | Reserved except for | OOOOb - No Error 
the following OOxxb - PCU internal error 
23:20 | Reserved Reserved 
31:24 | Reserved except for | 00h - No Error 
the following O9h - MC_MESSAGE_CHANNEL_TIMEOQUT 


13h - MC_DMI_TRAINING_TIMEOUT 

15h - MC_DMI_CPU_RESET_ACK_TIMEOUT 

1€h - MC_VR_ICC_MAX_LT_FUSED_ICC_MAX 
25h - MC_SVID_COMMAND_TIMEOUT 

29h - MC_VR_VOUT_MAC_LT_FUSED_SVID 

2Bh - MC_PKGC_WATCHDOG_HANG_CBZ_DOWN 
2Ch - MC_PKGC_WATCHDOG_HANG_CBZ_UP 
44h - MC_CRITICAL_VR_FAILED 

46h - MC_VIDLRAMP_DOWN_FAILED 

49h - MC_SVID_WRITE_REG_VOUT_MAX_FAILED 
4Bh - MC_BOOT_VID_TIMEOUT. Timeout setting boot VID for DRAM 0. 
4Fh - MC_SVID_COMMAND_ERROR. 

52h - MC_FIVR_CATAS_OVERVOL_FAULT. 

53h - MC_FIVR_CATAS_OVERCUR_FAULT. 

57h - MC_SVID_PKGC_REQUEST_FAILED 

58h - MC_SVID_IMON_REQUEST_FAILED 

59h - MC_SVID_ALERT_REQUEST_FAILED 

62h - MC_INVALID_PKGS_RSP_QPI 

64h - MC_INVALID_PKG_STATE_CONFIG 

67h - MC_HA_IMC_RW_BLOCK_ACK_TIMEOQUT 
6Ah - MC_MSGCH_PMREQ_CMP_TIMEOQUT 

72h - MC_WATCHDG_TIMEOUT_PKGS_MASTER 
81h - MC_RECOVERABLE_DIE_THERMAL_TOO_HOT 


56:32 | Reserved Reserved 
Status register 63:57 
validity indicators? 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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2. The internal error codes may be model-specific. 


16.6.2 Intel QPI Machine Check Errors 


MC error codes associated with the Intel QPI agents are reported in the MSRs IA32_MC5_STATUS, 
IA32_MC20_STATUS, and IA32_MC21_STATUS. The supported error codes follow the architectural MCACOD defi- 
nition type 1PPTRRRRIILL (see Chapter 15, “Machine-Check Architecture,”). 


Table 16-21 lists model-specific fields to interpret error codes applicable to IA32_MC5_STATUS, 
IA32_MC20_STATUS, and IA32_MC21_ STATUS. 


Table 16-21. Intel QPI MC Error Codes for IA32_MCi_STATUS (i = 5, 20, 21) 


Type BitNo. | Bit Function Bit Description 

MCA error 15:0 MCACOD Bus error format: 1PPTRRRRIILL 

codes! 

Model specific | 31:16 | MSCOD O2h - Intel QPI physical layer detected drift buffer alarm. 
errors 03h - Intel QPI physical layer detected latency buffer rollover. 


10h - Intel QPI link layer detected control error from R3QPI. 

11h - Rx entered LLR abort state on CRC error. 

12h - Unsupported or undefined packet. 

13h - Intel QPI link layer control error. 

15h - RBT used un-initialized value. 

20h - Intel QPI physical layer detected a QPI in-band reset but aborted initialization 
21h - Link failover data self-healing 

22h - Phy detected in-band reset (no width change). 

23h - Link failover clock failover 

30h -Rx detected CRC error - successful LLR after Phy re-init. 
31h -Rx detected CRC error - successful LLR without Phy re-init. 
All other values are reserved. 


37:32 | Reserved Reserved 
52:38 | Corrected Error Cnt 
56:53 | Reserved Reserved 
Status register | 63:57 
validity 
indicators? 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.6.3. Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC9_STATUS- 
IA32_MC1i6_STATUS. The supported error codes follow the architectural MCACOD definition type IMMMCCCC (see 
Chapter 15, “Machine-Check Architecture,”). 


MSR_ERROR_CONTROL. [bit 1] can enable additional information logging of the IMC. The additional error informa- 
tion logged by the IMC is stored in IA32_MCi_STATUS and IA32_MCi_MISC; (i = 9-16). 


IA32_MCi_STATUS (i=9-12) log errors from the first memory controller. The second memory controller logs into 
IA32_MCi_STATUS (i=13-16). 
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Table 16-22. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 9-16) 


Type 


Bit No. 


Bit Function 


Bit Description 


MCA error codes! 


15:0 


MCACOD 


Memory Controller error format: O000 0000 1MMM CCCC 


Model specific 
errors 


31:16 


the following 


Reserved except for 


0001H - DDR3 address parity error 
0002H - Uncorrected HA write data error 


O004H - Uncorrected HA data byte enable error 
OOO8H - Corrected patrol scrub error 

0010H - Uncorrected patrol scrub error 

0020H - Corrected spare error 

0040H - Uncorrected spare error 


OO80H - Corrected memory read error. (Only applicable with iMC’s “Additional 
Error logging” Mode-1 enabled.) 


0100H - iMC, write data buffer parity errors 
0200H - DDR4 command address parity error 


36:32 


Other info 


When MSR_ERROR_CONTROL [1] is set, logs an encoded value from the first error 
device. 


37 
56:38 


Reserved 


Reserved 
See Chapter 15, “Machine-Check Architecture,” 


Status register 
validity indicators? 


63:57 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


Table 16-23. Intel IMC MC Error Codes for IA32_MCi_MISC (i= 9-16) 


Type BitNo. | Bit Function Bit Description 
MCA addr info! | 8:0 See Chapter 15, “Machine-Check Architecture,” 
Model specific | 13:9 If the error logged is MCWrDataPar error or MCWrBEPar error, this field is the WDB 
errors ID that has the parity error. OR if the second error logged is a correctable read 
error, MC logs the second error device in this field. 
Model specific | 29:14 | ErrMask_1st€rrDev When MSR_ERROR_CONTROL|[1] is set, allows the iMC to log first-device error bit 
errors mask. 
Model specific | 45:30 | ErrMask_2nd€rrDev | When MSR_ERROR_CONTROL|[1] is set, allows the iMC to log second-device error 
errors bit mask. 
50:46 | FailRank_1st€rrDev | When MSR_ERROR_CONTROL|1] is set, allows the iMC to log first-device error 
failing rank. 
55:51 ‘| FailRank_2ndE€rrDev | When MSR_ERROR_CONTROL|1] is set, allows the iMC to log second-device error 
failing rank. 
61:56 Reserved 
62 Valid_1st€rrDev When MSR_ERROR_CONTROL|1] is set, indicates the iMC has logged valid data 
from a correctable error from memory read associated with first error device. 
63 Valid_2ndErrDev When MSR_ERROR_CONTROL,[1] is set, indicates the iMC has logged valid data due 
to a second correctable error in a memory device. Use this information only after 
there is valid first error info indicated by bit 62. 
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NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.6.4 |Home Agent Machine Check Errors 


Memory errors from the first memory controller may be logged in the IA32_MC7_{STATUS,ADDR, MISC} registers, 
while the second memory controller logs to IA32_MC8_{STATUS,ADDR,MISC}. 


16.7. INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR D 
FAMILY, MACHINE ERROR CODES FOR MACHINE CHECK 


The Intel® Xeon® processor D family is based on the Broadwell microarchitecture and can be identified with CPUID 
DisplayFamily_DisplaySignature 06_56H. Incremental error codes for internal machine check error from PCU 
controller is reported in the register bank IA32_MC4, Table 16-24 lists model-specific fields to interpret error codes 
applicable to IA32_MC4_STATUS. Incremental error codes for the memory controller unit is reported in the register 
banks IA32_MC9-IA32_MC10. Table 16-18 lists model-specific error codes apply to IA32_MCi_STATUS, i = 9-10. 


16.7.1 Internal Machine Check Errors 


Table 16-24. Machine Check Error Codes for [A32_MC4_STATUS 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD 
MCACOD2 15:0 internal Errors 0402h - PCU internal Errors 


0403h - internal Errors 
0406h - Intel TXT Errors 
0407h - Other UBOX internal Errors. 


On an IERR caused by a core 3-strike the [A32_MC3_STATUS (MLC) is copied 
to the IA32_MC4_STATUS (After a 3-strike, the core MCA banks will be 
unavailable). 


Model specific errors | 19:16 | Reserved except for | OOOOb - No Error 
the following 00x1b - PCU internal error 
001xb - PCU internal error 
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Type 


Bit No. 


Bit Function 


Bit Description 


23:20 


Reserved except for 
the following 


X1xxb - UBOX error 


31:24 


Reserved except for 
the following 


OOh - No Error 

O9h - MC_MESSAGE_CHANNEL_TIMEOQUT 

13h - MC_DMI_TRAINING_TIMEOUT 

15h - MC_DMI_CPU_RESET_ACK_TIMEOUT 
1€h - MC_VR_ICC_MAX_LT_FUSED_ICC_MAX 
25h - MC_SVID_COMMAND_TIMEOUT 

26h - MCA_PKGC_DIRECT_WAKE_RING_TIMEOUT 
29h - MC_VR_VOUT_MAC_LT_FUSED_SVID 

2Bh - MC_PKGC_WATCHDOG_HANG_CBZ_DOWN 
2Ch - MC_PKGC_WATCHDOG_HANG_CBZ_UP 
44h - MC_CRITICAL_VR_FAILED 

46h - MC_VID_RAMP_DOWN_FAILED 

49h - MC_SVID_WRITE_REG_VOUT_MAX_FAILED 


4Bh - MC_PP1_BOOT_VID_TIMEOUT. Timeout setting boot VID for DRAM 0. 
4Fh - MC_SVID_COMMAND_ERROR. 

52h - MC_FIVR_CATAS_OVERVOL_FAULT. 

53h - MC_FIVR_CATAS_OVERCUR_FAULT. 

57h - MC_SVID_PKGC_REQUEST_FAILED 

58h - MC_SVID_IMON_REQUEST_FAILED 

59h - MC_SVID_ALERT_REQUEST_FAILED 

62h - MC_INVALID_PKGS_RSP_QPI 

64h - MC_INVALID_PKG_STATE_CONFIG 

67h - MC_HA_IMC_RW_BLOCK_ACK_TIMEOUT 

6Ah - MC_MSGCH_PMREQ_CMP_TIMEOQUT 

72h - MC_WATCHDG_TIMEOUT_PKGS_MASTER 

81h - MC_RECOVERABLE_DIE_THERMAL_TOO_HOT 


56:32 


Reserved 


Reserved 


Status register 
validity indicators 


1 


63:57 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
2. The internal error codes may be model-specific. 


16.7.2 Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC9_STATUS- 
IA32_MC10_STATUS. The supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see 
Chapter 15, “Machine-Check Architecture,”). 


MSR_ERROR_CONTROL.[bit 1] can enable additional information logging of the IMC. The additional error informa- 
tion logged by the IMC is stored in IA32_MCi_STATUS and IA32_MCi_MISC; (i = 9-10). 
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Table 16-25. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 9-10) 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Memory Controller error format: 0000 0000 1MMM CCCC 
Model specific 31:16 | Reserved except for | 0001H - DDR3 address parity error 
errors the following 0002H - Uncorrected HA write data error 
QO04H - Uncorrected HA data byte enable error 
OOO8H - Corrected patrol scrub error 
0010H - Uncorrected patrol scrub error 
0100H - iMC, write data buffer parity errors 
0200H - DDR4 command address parity error 
36:32 | Other info Reserved 
37 Reserved Reserved 
56:38 See Chapter 15, “Machine-Check Architecture,” 
Status register 63:57 
validity indicators? 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR E5 
V4 FAMILY, MACHINE ERROR CODES FOR MACHINE CHECK 


The Intel® Xeon® processor E5 v4 family is based on the Broadwell microarchitecture and can be identified with 
CPUID DisplayFamily_DisplaySignature 06_4FH. Incremental error codes for internal machine check error from 
PCU controller is reported in the register bank IA32_MC4, Table 16-20 in Section 16.6. 1lists model-specific fields 
to interpret error codes applicable to IA32_MC4_STATUS. 


Incremental MC error codes related to the Intel QPI links are reported in the register banks IA32_MC5, 
IA32_MC20, and IA32_MC21. Information listed in Table 16-21 of Section 16.6.1 covers QPI MC error codes. 


16.8 


16.8.1. Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC9_STATUS- 
IA32_MC1i6_STATUS. The supported error codes follow the architectural MCACOD definition type I1MMMCCCC (see 
Chapter 15, “Machine-Check Architecture”). 


Table 16-26 lists model-specific error codes apply to IA32_MCi_STATUS, i = 9-16. 


IA32_MCi_STATUS (i=9-12) log errors from the first memory controller. The second memory controller logs into 
IA32_MCi_STATUS (i=13-16). 
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Table 16-26. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 9-16) 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Memory Controller error format: O000 0000 1MMM CCCC 
Model specific 31:16 | Reserved except for | 0001H - DDR3 address parity error 
errors the following 0002H - Uncorrected HA write data error 
O004H - Uncorrected HA data byte enable error 
OOO8H - Corrected patrol scrub error 
0010H - Uncorrected patrol scrub error 
0020H - Corrected spare error 
0040H - Uncorrected spare error 
0100H - iMC, write data buffer parity errors 
0200H - DDR4 command address parity error 
36:32 | Other info Reserved 
37 Reserved Reserved 
56:38 See Chapter 15, “Machine-Check Architecture,” 
Status register 63:57 
validity indicators? 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.8.2 |Home Agent Machine Check Errors 


MC error codes associated with mirrored memory corrections are reported in the MSRs IA32_MC7_MISC and 
IA32_MC8_MISC. Table 16-27 lists model-specific error codes apply to IA32_MCi_MISC, i = 7, 8. 


Memory errors from the first memory controller may be logged in the IA32_MC7_{STATUS,ADDR, MISC} registers, 
while the second memory controller logs to IA32_MC8_{STATUS,ADDR,MISC}. 


Table 16-27. Intel HA MC Error Codes for [A32_MCi_MISC (i= 7, 8) 


Bit No. | Bit Function Bit Description 

5:0 LSB See Figure 15-8. 

8:6 Address Mode See Table 15-3. 

40:9 Reserved Reserved 

41 Failover Error occurred at a pair of mirrored memory channels. Error was corrected by mirroring with 
channel failover. 

42 Mirrorcorr Error was corrected by mirroring and primary channel scrubbed successfully. 

63:43 Reserved Reserved 

16.9 INCREMENTAL DECODING INFORMATION: INTEL® XEON® PROCESSOR 


SCALABLE FAMILY, MACHINE ERROR CODES FOR MACHINE CHECK 


In the Intel® Xeon® Processor Scalable Family with CPUID DisplayFamily_DisplaySignature 06_55H, incremental 
error codes for internal machine check errors from the PCU controller are reported in the register bank IA32_MC4. 
Table 16-28 in Section 16.9.1 lists model-specific fields to interpret error codes applicable to IA32_MC4_STATUS. 
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16.9.1. Internal Machine Check Errors 
Table 16-28. Machine Check Error Codes for |A32_MC4_STATUS 
Type Bit No. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD 
MCACOD? 15:0 Internal Errors 0402h - PCU internal Errors 
0403h - PCU internal Errors 
0406h - Intel TXT Errors 
0407h - Other UBOX internal Errors. 
On an IERR caused by a core 3-strike the [A32_MC3_STATUS (MLC) is copied 
to the IA32_MC4_STATUS (After a 3-strike, the core MCA banks will be 
unavailable). 
Model specific errors | 19:16 | Reserved except for | OO0Ob - No Error 
the following OOxxb - PCU internal error 
23:20 | Reserved Reserved 
31:24 | Reserved except for | OOh - No Error 


the following 


ODh - MCA_DMI_TRAINING_TIMEOUT 

OFh - MCA_DMI_CPU_RESET_ACK_TIMEOUT 
10h - MCA_LMORE_THAN_ONE_LT_AGENT 
1€h - MCA_BIOS_RST_CPL_INVALID_SEQ 
1Fh - MCA_BIOS_INVALID_PKG_STATE_CONFIG 
25h - MCA_MESSAGE_CHANNEL_TIMEOUT 
27h - MCA_MSGCH_PMREQ_CMP_TIMEOUT 
30h - MCA_PKGC_DIRECT_WAKE_RING_TIMEOQUT 
31h - MCA_PKGC_INVALID_RSP_PCH 

33h - MCA_PKGC_WATCHDOG_HANG_CBZ_DOWN 
34h - MCA_PKGC_WATCHDOG_HANG_CBZ_UP 
38h - MCA_PKGC_WATCHDOG_HANG_C3_UP_SF 
40h - MCA_SVID_VCCIN_VR_ICC_MAX_FAILURE 
41h - MCA_SVID_COMMAND_TIMEOUT 
42h - MCA_SVID_VCCIN_VR_VOUT_MAX_FAILURE 
43h - MCA_SVID_CPU_VR_CAPABILITY_ERROR 
44h - MCA_SVID_CRITICAL_VR_FAILED 

45h - MCA_SVID_SA_ITD_ERROR 

46h - MCA_SVID_LREAD_REG_FAILED 

47h - MCA_SVID_WRITE_REG_FAILED 

48h - MCA_SVID_PKGC_INIT_FAILED 
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Type BitNo. | Bit Function Bit Description 


49h - MCA_SVID_PKGC_CONFIG_FAILED 

4Ah - MCA_SVID_PKGC_REQUEST_FAILED 

4Bh - MCA_SVID_IMON_REQUEST_FAILED 

4Ch - MCA_SVID_ALERT_REQUEST_FAILED 

4Dh - MCA_SVID_MCP_VP_ABSENT_OR_RAMP_ERROR 
4€h - MCA_SVID_UNEXPECTED_MCP_VP_DETECTED 
51h - MCA_FIVR_CATAS_OVERVOL_FAULT 

52h - MCA_FIVR_CATAS_OVERCUR_FAULT 

58h - MCA_WATCHDG_TIMEOUT_PKGC_SECONDARY 
59h - MCA_WATCHDG_TIMEOUT_PKGC_MAIN 

5Ah - MCA_WATCHDG_TIMEOQUT_PKGS_MAIN 

61h - MCA_PKGS_CPD_UNPCD_TIMEOUT 

63h - MCA_PKGS_INVALID_REQ_PCH 

64h - MCA_PKGS_INVALID_REQ_INTERNAL 

65h - MCA_PKGS_INVALID_RSP_INTERNAL 

6Bh - MCA_PKGS_SMBUS_VPP_PAUSE_TIMEOUT 
81h - MC_RECOVERABLE_DIE_THERMAL_TOO_HOT 


52:32 | Reserved Reserved 
54:53 | CORR_ERR_STATUS _ | Reserved 
56:55 | Reserved Reserved 

Status register 63:57 

validity indicators? 

NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
2. The internal error codes may be model-specific. 


16.9.2 Interconnect Machine Check Errors 


MC error codes associated with the link interconnect agents are reported in the MSRs IA32_MC5_STATUS, 
IA32_MC12_STATUS, IA32_MC19_STATUS. The supported error codes follow the architectural MCACOD definition 
type 1PPTRRRRIILL (see Chapter 15, “Machine-Check Architecture”). 


Table 16-29 lists model-specific fields to interpret error codes applicable to IA32_MCi_STATUS, i= 5, 12, 19. 


Table 16-29. Interconnect MC Error Codes for I|A32_MCi_STATUS, i = 5, 12, 19 


Type BitNo. | Bit Function Bit Description 
MCA error 15:0 MCACOD Bus error format: 1PPTRRRRIILL 
codes The two supported compound error codes: 


- OxOCOF - Unsupported/Undefined Packet 
- OxOEOF - For all other corrected and uncorrected errors 
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Type 


Bit No. 


Bit Function 


Bit Description 


Model specific 
errors 


21:16 


MSCOD 


The encoding of Uncorrectable (UC) errors are: 

OOh - UC Phy Initialization Failure. 

01h - UC Phy detected drift buffer alarm. 

O2h - UC Phy detected latency buffer rollover. 

10h - UC link layer Rx detected CRC error: unsuccessful LLR entered abort state 
11h - UC LL Rx unsupported or undefined packet. 

12h - UC LL or Phy control error. 

13h - UC LL Rx parameter exchange exception. 

1fh - UC LL detected control error from the link-mesh interface 

The encoding of correctable (COR) errors are: 

20h - COR Phy initialization abort 

21h - COR Phy reset 

22h - COR Phy lane failure, recovery in x8 width. 

23h - COR Phy LOc error corrected without Phy reset 

24h - COR Phy LOc error triggering Phy reset 

25h - COR Phy LOp exit error corrected with Phy reset 

30h - COR LL Rx detected CRC error - successful LLR without Phy re-init. 
31h - COR LL Rx detected CRC error - successful LLR with Phy re-init. 

All other values are reserved. 


31:22 


MSCOD_SPARE 


The definition below applies to MSCOD 12h (UC LL or Phy Control Errors) 
[Bit 22]: Phy Control Error 

[Bit 23] : Unexpected Retry.Ack flit 

[Bit 24] : Unexpected Retry.Req flit 

[Bit 25] : RF parity error 

[Bit 26] : Routeback Table error 

[Bit 27] : unexpected Tx Protocol flit (EOP, Header or Data) 

[Bit 28] : Rx Header-or-Credit BGF credit overflow/underflow 


[Bit 29]: Link Layer Reset still in progress when Phy enters LO (Phy training should 
not be enabled until after LL reset is complete as indicated by 
KTILCL.LinkLayerReset going back to 0). 


[Bit 30] : Link Layer reset initiated while protocol traffic not idle 
[Bit 31] : Link Layer Tx Parity Error 


37:32 


Reserved 


Reserved 


52:38 
56:53 


Corrected Error Cnt 


Reserved 


Reserved 


Status register 
validity 


indicators? 


NOTES: 


63:57 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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16.9.3. Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC13_STATUS- 
IA32_MC18_STATUS. The supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see 
Chapter 15, “Machine-Check Architecture”). 


IA32_MCi_STATUS (i=13,14,17) log errors from the first memory controller. The second memory controller logs 
into IA32_MCi_STATUS (i=15,16,18). 


Table 16-30. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 13-18) 


Type Bit No. | Bit Function Bit Description 

MCA error codes! 15:0 MCACOD Memory Controller error format: O000 0000 1MMM CCCC 
Model specific 31:16 | Reserved except for | 0001H - Address parity error 

errors the following 0002H - HA write data parity error 


O0004H - HA write byte enable parity error 

OOO8H - Corrected patrol scrub error 

0010H - Uncorrected patrol scrub error 

0020H - Corrected spare error 

0040H - Uncorrected spare error 

OO80H - Any HA read error 

0100H - WDB read parity error 

0200H - DDR4 command address parity error 

0400H - Uncorrected address parity error 

O800H - Unrecognized request type 

0801H - Read response to an invalid scoreboard entry 
O0802H - Unexpected read response 

0803H - DDR4 completion to an invalid scoreboard entry 
0804H - Completion to an invalid scoreboard entry 
O805H - Completion FIFO overflow 

O806H - Correctable parity error 

0807H - Uncorrectable error 

O808H - Interrupt received while outstanding interrupt was not ACKed 
O809H - ERID FIFO overflow 

O80eH - Error on Write credits 

O8O0bH - Error on Read credits 

O80cH - Scheduler error 

O80dH - Error event 


36:32 | Other info MC logs the first error device. This is an encoded 5-bit value of the device. 
37 Reserved Reserved 
56:38 See Chapter 15, “Machine-Check Architecture,” 

Status register 63:57 


1 


validity indicators 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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16.9.4  M2M Machine Check Errors 


MC error codes associated with M2M are reported in the MSRs IA32_MC7_STATUS, IA32_MC8_STATUS. The 
supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see Chapter 15, “Machine- 
Check Architecture,”). 


Table 16-31. M2M MC Error Codes for IA32_MCi_STATUS (i= 7-8) 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Compound error format: 0000 0000 1MMM CCCC 
Model specific 16 MscodDataRdErr Logged an MC read data error 
errors 
17 Reserved Reserved 
18 MscodPtlWr€rr Logged an MC partial write data error 
19 MscodFullwr€rr Logged a full write data error 
20 MscodBaf€rr Logged an M2M clock-domain-crossing buffer (BGF) error 
21 MscodTimeOut Logged an M2M time out 
22 MscodPar€rr Logged an MeM tracker parity error 
23 MscodBucket1 Err Logged a fatal Bucket1 error 
31:24 | Reserved Reserved 
36:32 | Other info MC logs the first error device. This is an encoded 5-bit value of the device. 
37 Reserved Reserved 
56:38 See Chapter 15, “Machine-Check Architecture,” 
Status register 63:57 
validity indicators? 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.9.5 |Home Agent Machine Check Errors 


MC error codes associated with mirrored memory corrections are reported in the MSRs IA32_MC7_MISC and 
IA32_MC8_MISC. Table 16-32 lists model-specific error codes apply to IA32_MCi_MISC, i = 7, 8. 


Memory errors from the first memory controller may be logged in the IA32_MC7_{STATUS,ADDR, MISC} registers, 
while the second memory controller logs to IA32_MC8_{STATUS,ADDR,MISC}. 


Table 16-32. Intel HA MC Error Codes for [A32_MCi_MISC (i= 7, 8) 


Bit No. | Bit Function Bit Description 

5:0 LSB See Figure 15-8. 

8:6 Address Mode See Table 15-3. 

40:9 Reserved Reserved 

61:41 Reserved Reserved 

62 Mirrorcorr Error was corrected by mirroring and primary channel scrubbed successfully. 

63 Failover Error occurred at a pair of mirrored memory channels. Error was corrected by mirroring with 
channel failover. 
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16.10 INCREMENTAL DECODING INFORMATION: PROCESSOR FAMILY WITH CPUID 
DISPLAYFAMILY_DISPLAYMODEL SIGNATURE 06_5FH, MACHINE ERROR 
CODES FOR MACHINE CHECK 


In Intel® Atom® processors based on Goldmont Microarchitecture with CPUID DisplayFamily_DisplaySignature 
06_5FH (code name Denverton), incremental error codes for the memory controller unit are reported in the 
register banks IA32_MC6 and IA32_MC7. Table 16-33 in Section 16.10.1 lists model-specific fields to interpret 
error codes applicable to IA32_MCi_STATUS, i = 6, 7. 


16.10.1 Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC6_STATUS and 
IA32_MC7_STATUS. The supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see 
Chapter 15, “Machine-Check Architecture”). 


Table 16-33. Intel IMC MC Error Codes for IA32_MCi_STATUS (i= 6, 7) 


Type BitNo. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD 
Model specific errors | 31:16 | Reserved except for / 01h - Cmd/Addr parity 
the following O2h - Corrected Demand/Patrol Scrub Error 


04h - Uncorrected patrol scrub error 
O8h - Uncorrected demand read error 
10h - WDB read ECC 


36:32 | Other info 


37 Reserved 


56:38 See Chapter 15, “Machine-Check Architecture”. 
Status register 63:57 
validity indicators? 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.11 INCREMENTAL DECODING INFORMATION: FUTURE INTEL® XEON® 
PROCESSORS WITH CPUID DISPLAYFAMILY_DISPLAYMODEL SIGNATURES 
06_6AH AND 06_6CH, MACHINE ERROR CODES FOR MACHINE CHECK 


Future Intel® Xeon® processor with CPUID DisplayFamily_DisplaySignature of 06_6AH and 06_6CH, incremental 
error codes for internal machine check errors from the PCU controller are reported in the register bank IA32_MC4. 
Table 16-34 in Section 16.11.1 lists model-specific fields to interpret error codes applicable to IA32_MC4_STATUS. 
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Table 16-34. Machine Check Error Codes for [A32_MC4_STATUS 


Type 


Bit No. 


Bit Function 


Bit Description 


Machine Check Error 
Codes! 


15:0 


MCCOD 


MCCOD 


15:0 


Internal Errors 


The value of this field will be 0402h for the PCU and 0406h for internal 
firmware errors. 


This applies for any logged error. 


Model specific errors 


19:16 


Reserved except for the 
following 


Model specific error code bits 19:16. 


This logs the type of HW UC (PCU/VCU) error that has occurred. There are 
7 errors defined. 


01h - Instruction address out of valid space. 
02h - Double bit RAM error on Instruction Fetch. 
O3h - Invalid OpCode seen. 

04h - Stack Underflow. 

O5h - Stack Overflow. 

OGh - Data address out of valid space. 

07h - Double bit RAM error on Data Fetch. 


23:20 


Reserved except for the 
following 


Model specific error code bits 23:20. 


This logs the type of HW FSM error that has occurred. There are 3 errors 
defined. 


04h - Clock/power IP response timeout. 
05h - SMBus controller raised SMI. 
O9h - PM controller received invalid transaction. 


31:24 


Reserved except for the 
following 


ODh - MCA_LLC_BIST_ACTIVE_TIMEOUT 

OEh - MCA_DMI_TRAINING_TIMEOUT 

OFh - MCA_DMI_STRAP_SET_ARRIVAL_TIMEOUT 
10h - MCA_DMI_CPU_RESET_ACK_TIMEOUT 
11h - MCA_LMORE_THAN_ONE_LT_AGENT 

14h - MCA_INCOMPATIBLE_PCH_TYPE 
1€h - MCA_BIOS_RST_CPL_INVALID_SEQ 
1Fh - MCA_BIOS_INVALID_PKG_STATE_CONFIG 
2Dh - MCA_PCU_PMAX_CALIB_ERROR 

2Eh - MCA_TSC100_SYNC_TIMEOUT 

3Ah - MCA_GPSB_TIMEOUT 

3Bh - MCA_PMSB_TIMEOUT 

3Eh - MCA_IOSFSB_PMREQ_CMP_TIMEOQUT 
40h - MCA_SVID_VCCIN_VR_ICC_MAX_FAILURE 
42h - MCA_SVID_VCCIN_VR_VOUT_FAILURE 
43h - MCA_SVID_CPU_VR_CAPABILITY_ERROR 
44h - MCA_SVID_CRITICAL_VR_FAILED 

45h - MCA_SVID_SA_ITD_ERROR 

46h - MCA_SVID_LREAD_REG_FAILED 
47h - MCA_SVID_WRITE_REG_FAILED 

4Ah - MCA_SVID_PKGC_REQUEST_FAILED 
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Type 


Bit No. 


Bit Function 


Bit Description 


4Bh - MCA_SVID_IMON_REQUEST_FAILED 

4Ch - MCA_SVID_ALERT_REQUEST_FAILED 

4Dh - MCA_SVID_MCP_VR_RAMP_ERROR 

56h - MCA_FIVR_PD_HARDERR 

58h - MCA_WATCHDOG_TIMEOUT_PKGC_SECONDARY 
59h - MCA_WATCHDOG_TIMEOUT_PKGC_MAIN 
5Ah - MCA_WATCHDOG_TIMEOUT_PKGS_MAIN 
5Bh - MCA_WATCHDOG_TIMEOUT_MSG_CH_FSM 
5Ch - MCA_WATCHDOG_TIMEOUT_BULK_CR_FSM 
5Dh - MCA_WATCHDOG_TIMEOUT_IOSFSB_FSM 
60h - MCA_PKGS_SAFE_WP_TIMEOUT 

61h - MCA_PKGS_CPD_UNCPD_TIMEOUT 

62h - MCA_PKGS_INVALID_REQ_PCH 

63h - MCA_PKGS_INVALID_REQ_INTERNAL 

64h - MCA_PKGS_INVALID_RSP_INTERNAL 
65h-7Ah - MCA_PKGS_RESET_PREP_TIMEOUT 
7Bh - MCA_PKGS_SMBUS_VPP_PAUSE_TIMEQUT 
7Ch - MCA_PKGS_SMBUS_MCP_PAUSE_TIMEOUT 
7Dh - MCA_PKGS_SMBUS_SPD_PAUSE_TIMEOUT 
80h - MCA_PKGC_DISP_BUSY_TIMEOUT 

81h - MCA_PKGC_INVALID_RSP_PCH 

83h - MCA_PKGC_WATCHDOG_HANG_CBZ_DOWN 
84h - MCA_PKGC_WATCHDOG_HANG_CBZ_UP 
87h - MCA_PKGC_WATCHDOG_HANG_C2_BLKMASTER 
88h - MCA_PKGC_WATCHDOG_HANG_C2_PSLIMIT 
89h - MCA_PKGC_WATCHDOG_HANG_SETDISP 
8Bh - MCA_PKGC_ALLOW_L1_ERROR 

90h - MCA_RECOVERABLE_DIE_THERMAL_TOO_HOT 
AOh - MCA_ADR_SIGNAL_TIMEOUT 

Aih- MCA_BCLK_FREQ_OC_ABOVE_THRESHOLD 
BOh - MCA_DISPATCHER_RUN_BUSY_TIMEOUT 


37:32 


ENH_MCA_AVAILO 


Available when Enhanced MCA is in use. 


52:38 
54:53 


CORR_ERR_COUNT 
CORRERRORSTATUSIND 


Correctable error count. 


These bits are used to indicate when the number of corrected errors has 
exceeded the safe threshold to the point where an uncorrected error has 


become more likely to happen. 
Table 3 shows the encoding of these bits. 


56:55 


ENH_MCA_AVAIL1 


Available when Enhanced MCA is in use. 


Status register 
validity indicators 


1 


63:57 


NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16-30 Vol. 3B 


INTERPRETING MACHINE-CHECK ERROR CODES 


16.11.2 Interconnect Machine Check Errors 


MC error codes associated with the link interconnect agents are reported in the MSRs IA32_MC5_STATUS, 
IA32_MC7_STATUS, IA32_MC8_STATUS. The supported error codes follow the architectural MCACOD definition 
type 1PPTRRRRIILL (see Chapter 15, “Machine-Check Architecture”). 


NOTE 


The interconnect machine check errors in this section apply only to future Intel Xeon processors 
with a CPUID DisplayFamily_DisplaySignature of 06_6AH. These do not apply to future Intel Xeon 
processors with a CPUID DisplayFamily_DisplaySignature of 06_6CH. 


Table 16-35 lists model-specific fields to interpret error codes applicable to IA32_MCi_STATUS, i= 5, 7, 8. 
Table 16-35. Interconnect MC Error Codes for IA32_MCi_STATUS, i= 5, 7, 8 


Type BitNo. | Bit Function Bit Description 
MCA error 15:0 MCACOD Bus error format: 1PPTRRRRIILL 
codes The two supported compound error codes: 


- OxOCOF - Unsupported/Undefined Packet. 
- OxOEOF - For all other corrected and uncorrected errors. 


Model specific | 21:16 | MSCOD The encoding of Uncorrectable (UC) errors are: 

errors OOh - Phy Initialization Failure (NumInit). 

01h - Phy Detected Drift Buffer Alarm. 

O2h - Phy Detected Latency Buffer Rollover. 

10h - LL Rx detected CRC error: unsuccessful LLR (entered Abort state). 
11h - LL Rx Unsupported/Undefined packet. 

12h - LLor Phy Control Error. 

13h - LL Rx Parameter Exception. 

1Fh - LL Detected Control Error. 

The encoding of correctable (COR) errors are: 

20h - Phy Initialization Abort. 

21h - Phy Inband Reset. 

22h - Phy Lane failure, recovery in x8 width. 

23h - Phy LOc error corrected without Phy reset. 

24h - Phy LOc error triggering Phy reset. 

25h - Phy LOp exit error corrected with reset. 

30h - LL Rx detected CRC error: successful LLR without Phy Reinit. 
31h - LL Rx detected CRC error: successful LLR with Phy Reinit. 
32h - Tx received LLR. 

All other values are reserved. 
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Type Bit No. | Bit Function Bit Description 


31:22 | MSCOD_SPARE The definition below applies to MSCOD 12h (UC LL or Phy Control Errors). 
[Bit 22] : Phy Control Error. 

[Bit 23] : Unexpected Retry.Ack flit. 

[Bit 24] : Unexpected Retry.Req flit. 

[Bit 25] : RF parity error. 

[Bit 26] : Routeback Table error. 

[Bit 27] : Unexpected Tx Protocol flit (EOP, Header or Data). 

[Bit 28] : Rx Header-or-Credit BGF credit overflow/underflow. 


[Bit 29]: Link Layer Reset still in progress when Phy enters LO (Phy training should 
not be enabled until after LL reset is complete as indicated by 
KTILCL.LinkLayerReset going back to 0). 


[Bit 30] : Link Layer reset initiated while protocol traffic not idle. 
[Bit 31] : Link Layer Tx Parity Error. 

37:32 | OTHER_INFO Other Info. 

56:38 | Corrected Error Cnt See Chapter 15, “Machine-Check Architecture”. 


Status register | 63:57 
validity 
indicators 


1 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


16.11.3 Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers for the 3rd generation Intel® Xeon® Processor Scal- 
able Family based on Ice Lake microarchitecture are defined in Table 16-37. 


The MSRs reporting MC error codes differ depending on the CPUID DisplayFamily_DisplaySignature of the 
processor. See Table 16-36 for details. 


Table 16-36. MSRs Reporting MC Error Codes by CPUID DisplayFamily_DisplaySignature 


Processor CPUID MSRs Reporting MC Error Codes 
DisplayFamily_DisplaySignature 

3rd generation Intel® Xeon® Processor Scalable | O6_6AH IA32_MC13_STATUS - |A32_MC15_STATUS 

Family based on Ice Lake microarchitecture IA32_MC17_STATUS - |A32_MC19_ STATUS 


IA32_MC21_STATUS - |A32_MC23_STATUS 
IA32_MC25_STATUS - 1IA32_MC27_STATUS 
3rd generation Intel® Xeon® Processor Scalable | O6_6CH IA32_MC13_STATUS - IA32_MC15_STATUS 
Family based on Ice Lake microarchitecture 1A32_MC1 7_STATUS . 1A32_MC1 9 STATUS 


The supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see Chapter 15, “Machine- 
Check Architecture”). 
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Table 16-37. Intel IMC MC Error Codes for |A32_MCi_STATUS (i= 13-15, 17-19, 21-23, 25-27) 


Type BitNo. | Bit Function Bit Description 

MCA error codes! 15:0 MCACOD Memory Controller error format: 0000 0000 1MMM CCCC 
Model specific 31:16 | Reserved except for | 0001H - Address parity error. 

EILOIS the following O002H - Data parity error. 


0003H - Data ECC error. 

O004H - Data byte enable parity error. 

O0007H - Transaction ID parity error. 

OOO8H - Corrected patrol scrub error. 

0010H - Uncorrected patrol scrub error. 
O0020H - Corrected spare error. 

0040H - Uncorrected spare error. 

OO80H - Corrected read error. 

OOAOH - Uncorrected read error. 

OOCOH - Uncorrected metadata. 

0100H - WDB read parity error. 

0103H - RPA parity error. 

0104H - RPA parity error. 

0105H - WPA parity error. 

0106H - DDR_T_DPPP data BE error. 

0107H - DDR_T_DPPP data error. 

0108H - DDR link failure. 

0111H - PCLS CAM error. 

0112H - PCLS data error. 

0200H - DDR4 command / address parity error. 
0220H - HBM command / address parity error. 
0221H - HBM data parity error. 

0400H - RPQ parity (primary) error. 

0401H - RPQ parity (buddy) error. 

0404H - WPQ parity (primary) error. 

0405H - WPQ parity (buddy) error. 

0408H - RPB parity (primary) error. 

O409H - RPB parity (buddy) error. 

O800H - DDR-T bad request. 

0801H - DDR Data response to an invalid entry. 
O0802H - DDR data response to an entry not expecting data. 
0803H - DDR4 completion to an invalid entry. 
0804H - DDR-T completion to an invalid entry. 
O805H - DDR data/completion FIFO overflow. 
O806H - DDR-T ERID correctable parity error. 
0807H - DDR-T ERID uncorrectable error. 
O808H - DDR-T interrupt received while outstanding interrupt was not ACKed. 
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Type Bit No. | Bit Function Bit Description 


O809H - ERID FIFO overflow. 

O80AH - DDR-T error on FNV write credits. 
O80BH - DDR-T error on FNV read credits. 
O80CH -DDR-T scheduler error. 

O80DH - DDR-T FNV error event. 

O80EH - DDR-T FNV thermal event. 
O80FH - CMI packet while idle. 

0810H - DDR_T_RPQ_REQ_PARITY_ERR. 
0811H - DDR_T_WPQ_REQ_PARITY_ERR. 
0812H - 2LM_NMFILLWR_CAM_ERR. 
0813H - CMI_CREDIT_OVERSUB_ERR. 
0814H - CMI_CREDIT_TOTAL_ERR. 
0815H - CMI_CREDIT_RSVD_POOL_ERR. 
0816H - DDR_T_RD_ERROR. 

0817H - WDB_FIFO_ERR. 

0818H - CMI_LREQ_FIFO_OVERFLOW. 
0819H - CMI_REQ_FIFO_UNDERFLOW. 
081AH - CMI_RSP_FIFO_OVERFLOW. 
081BH - CMI_RSP_FIFO_UNDERFLOW. 
081CH - CMI_LMISC_MC_CRDT_ERRORS. 
081DH - CMI_LMISC_MC_ARB_ERRORS. 
081€H - DDR_T_WR_CMPL_FIFO_OVERFLOW. 
081FH - DDR_T_WR_CMPL_FIFO_UNDERFLOW. 
0820H - CMI_RD_CPL_FIFO_OVERFLOW. 
0821H - CMI_LRD_CPL_FIFO_UNDERFLOW. 
0822H - TME_KEY_PAR_ERR. 

0823H - TME_CMI_MISC_ERR. 

0824H - TME_CMI_OVFL_ERR. 

0825H - TME_CMI_UFL_ERR. 

0826H - TME_TEM_SECURE_ERR. 

0827H - TME_UFILL_PAR_ERR. 


37:32 | Other info Other Info. 
56:38 See Chapter 15, “Machine-Check Architecture”. 
Status register 63:57 
validity indicators2 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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Additional information is reported in the MSRs IA32_MC13_MISC —IA32_MC15_MISC, IA32_MC17_MISC — 
IA32_MC19_MISC, IA32_MC21_MISC — IA32_MC23_MISC, and IA32_MC25_MISC — IA32_MC27_MISC. Table 
16-38 lists the information reported in IA32_MCi_MISC, i = 13-15, 17-19, 21-23, and 25-27. 


Table 16-38. Additional Information Reported in [A32_MCi_MISC (i= 13-15, 17-19, 21-23, 25-27) 


Bit No. | Bit Function Bit Description 

5:0 LSB See Figure 15-8. 

8:6 Address Mode See Table 15-3. 

18:9 Column Component of sub-DIMM address. 


Bits 18-17: Reserved 

Bit 16: Column 9 

Bit 15: Column 8 

Bit 14: Column 7 

Bit 13: Column 6 

Bit 12: Column 5 

Bit 11: Column 4 

Bit 10: Column 3 

Bit 9: Reserved 

39:19 Row Component of sub-DIMM address. 
45:40 Bank Component of sub-DIMM address. 
Bit 45: Reserved 

Bit 44: Bank group 2 

Bit 43: Bank address 1 

Bit 42: Bank address O 

Bit 41: Bank group 1 

Bit 40: Bank address 0 


51:46 Failed Device Failing device for correctable error (not valid for uncorrectable or transient errors). 
55:52 SubRank Logical Rank (3D Stacked SDRAM) 
58:56 Rank Rank 
62:59 ECC Mode 0000b: SDDC memory mode 
0001b: SDDC 


0100b: ADDDC memory mode 
0101b: ADDDC 

1000b: Read from DDRT 
Other values: Reserved 

63 Transient Ob: 

1b: Error was transient 


16.11.4 M2M Machine Check Errors 


MC error codes associated with M2M for future Intel Xeon processors with a CPUID DisplayFamily_DisplaySignature 
of 06_6AH are reported in the MSRs IA32_MC12_STATUS, IA32_MC16_STATUS, IA32_MC20_STATUS, and 
IA32_MC24_STATUS. 


MC error codes associated with M2M for future Intel Xeon processors with a CPUID DisplayFamily_DisplaySignature 
of 06_6AH are reported in the MSRs IA32_MC12_STATUS and IA32_MC16_STATUS. 
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The supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see Chapter 15, “Machine- 


Check Architecture”). 


Table 16-39. M2M MC Error Codes for IA32_MCi_STATUS (i= 12, 16, 20, 24) 


Type Bit No. | Bit Function Bit Description 
MCA error codes! 15:0 MCACOD Compound error format: 0000 0000 1MMM CCCC 
Model specific 16 MscodDataRdErr Logged an MC read data error. 
errors 
17 Reserved Reserved 
18 MscodPtlwr€rr Logged an MC partial write data error. 
19 MscodFullwr€rr Logged a full write data error. 
20 MscodBaf€rr Logged an M2M clock-domain-crossing buffer (BGF) error. 
21 MscodTimeOut Logged an M2M time out. 
22 MscodPar€rr Logged an M2M tracker parity error. 
23 MscodBucket1 Err Logged a fatal Bucket1 error. 
25:24 | MscodDDRType Logged a DDR/DDR-T specific error. 
31:26 | MscodMisc€rrs Logged a miscellaneous error. 
37:32 | Other info Other Info. 
56:38 See Chapter 15, “Machine-Check Architecture”. 
Status register 63:57 
validity indicators? 


NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


MC error codes associated with mirrored memory corrections are reported in the MSRs IA32_MC12_MISC, 
IA32_MC16_MISC, IA32_MC20_MISC, and IA32_MC24_MISC. The model-specific error codes listed in Table 16-32 
also apply to IA32_MCi_MISC, i = 12, 16, 20, 24. 


INCREMENTAL DECODING INFORMATION: PROCESSOR FAMILY WITH CPUID 
DISPLAYFAMILY_DISPLAYMODEL SIGNATURE 06_86H, MACHINE ERROR 
CODES FOR MACHINE CHECK 


In Intel® Atom® processors based on Tremont microarchitecture with CPUID DisplayFamily_DisplaySignature 
06_86H, incremental error codes for internal machine check errors from the PCU controller are reported in the 
register bank IA32_MC4. Table 16-34 in Section 16.11.1 lists model-specific fields to interpret error codes appli- 
cable to IA32_MC4_STATUS. 


16.12 


16.12.1 Integrated Memory Controller Machine Check Errors 


MC error codes associated with integrated memory controllers are reported in the MSRs IA32_MC13_STATUS - 
IA32_MC15_STATUS. The supported error codes follow the architectural MCACOD definition type 1MMMCCCC (see 
Chapter 15, “Machine-Check Architecture”). 


The IA32_MCi_STATUS MSR (where i = 13, 14, 15) contains information related to a machine check error if its 
VAL(valid) flag is set. Bit definitions are the same as those found in Table 16-37 “Inte! IMC MC Error Codes for 
IA32_MCi_STATUS (i= 13-15, 17-19, 21-23, 25-27)”. 
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The IA32_MCi_MISC MSR (where i = 13, 14, 15) contains information related memory corrections. Bit definitions 
are the same as those found in Table 16-38 “Additional Information Reported in IA32_MCi_MISC (i= 13-15, 17-19, 
21-23, 25-27)". 


16.12.2 M2M Machine Check Errors 


MC error codes associated with M2M are reported in the IA32_MC12_STATUS MSR. The supported error codes 
follow the architectural MCACOD definition type 1MMMCCCC (see Chapter 15, “Machine-Check Architecture”). 


Bit definitions are the same as those found in Table 16-39 “M2M MC Error Codes for IA32_MCi_STATUS (i= 12, 16, 
20, 24)”. 


16.13. INCREMENTAL DECODING INFORMATION: PROCESSOR FAMILY OFH 


MACHINE ERROR CODES FOR MACHINE CHECK 


Table 16-40 provides information for interpreting additional family OFH model-specific fields for external bus 
errors. These errors are reported in the IA32_MCi_STATUS MSRs. They are reported architecturally) as compound 
errors with a general form of 0000 1PPT RRRR IILL in the MCA error code field. See Chapter 15 for information on 
the interpretation of compound error codes. 


Table 16-40. Incremental Decoding Information: Processor Family OFH Machine Error Codes For Machine Check 


Type Bit No. | Bit Function Bit Description 
MCA error 15:0 
codes! 
Model-specific | 16 FSB address parity Address parity error detected: 
error codes 1 = Address parity error detected 
O = No address parity error 
17 Response hard fail Hardware failure detected on response 
18 Response parity Parity error detected on response 
19 PIC and FSB data parity Data Parity detected on either PIC or FSB access 
20 Processor Signature = Processor Signature = OOOOOFO4H. Indicates error due to an invalid PIC 
OOOOOFOAH: Invalid PIC request access was made to PIC space with WB memory): 
request 1 = Invalid PIC request error 
O = No Invalid PIC request error 
Reserved 
All other processors: 
Reserved 
21 Pad state machine The state machine that tracks P and N data-strobe relative timing has 
become unsynchronized or a glitch has been detected. 
22 Pad strobe glitch Data strobe glitch 
23 Pad address glitch Address strobe glitch 
Other 56:24 | Reserved Reserved 
Information 
Status register | 63:57 
validity 
indicators? 
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NOTES: 
1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 


Table 16-10 provides information on interpreting additional family OFH, model specific fields for cache hierarchy 
errors. These errors are reported in one of the IA32_MCi_STATUS MSRs. These errors are reported, architecturally, 
as compound errors with a general form of QO00 0001 RRRR TTLL in the MCA error code field. See Chapter 15 for 
how to interpret the compound error code. 


16.13.1 Model-Specific Machine Check Error Codes for Intel Xeon Processor MP 7100 Series 


Intel Xeon processor MP 7100 series has 5 register banks which contains information related to Machine Check 
Errors. MCi_STATUS[63:0] refers to all 5 register banks. MCO_STATUS[63:0] through MC3_STATUS[63:0] is the 
same as on previous generation of Intel Xeon processors within Family OFH. MC4_STATUS[63:0] is the main error 
logging for the processor’s L3 and front side bus errors. It supports the L3 Errors, Bus and Interconnect Errors 
Compound Error Codes in the MCA Error Code Field. 


Table 16-41. MCi_STATUS Register Bit Definition 
Bit Field Name | Bits Description 


MCA_Error_Code | 15:0 Specifies the machine check architecture defined error code for the machine check error condition 
detected. The machine check architecture defined error codes are guaranteed to be the same for all 
Intel Architecture processors that implement the machine check architecture. See tables below 


Model_Specific_E | 31:16 | Specifies the model specific error code that uniquely identifies the machine check error condition 
rror_Code detected. The model specific error codes may differ among Intel Architecture processors for the same 
Machine Check Error condition. See tables below 


Other_Info 56:32 | The functions of the bits in this field are implementation specific and are not part of the machine check 
architecture. Software that is intended to be portable among Intel Architecture processors should not 
rely on the values in this field. 


PCC 57 Processor Context Corrupt flag indicates that the state of the processor might have been corrupted by 
the error condition detected and that reliable restarting of the processor may not be possible. When 
clear, this flag indicates that the error did not affect the processor's state. This bit will always be set for 
MC errors which are not corrected. 


ADDRV 58 MC_ADDR register valid flag indicates that the MC_ADDR register contains the address where the error 
occurred. When clear, this flag indicates that the MC_ADDR register does not contain the address where 
the error occurred. The MC_ADDR register should not be read if the ADDRV bit is clear. 


MISCV 59 MC_MISC register valid flag indicates that the MC_MISC register contains additional 

information regarding the error. When clear, this flag indicates that the MC_MISC register does not 
contain additional information regarding the error. MC_MISC should not be read if the MISCV bit is not 
set. 


EN 60 Error enabled flag indicates that reporting of the machine check exception for this error was enabled by 
the associated flag bit of the MC_CTL register. Note that correctable errors do not have associated 
enable bits in the MC_CTL register so the EN bit should be clear when a correctable error is logged. 
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Table 16-41. MCi_STATUS Register Bit Definition (Contd.) 


Bit Field Name | Bits Description 

UC 61 Error uncorrected flag indicates that the processor did not correct the error condition. When clear, this 
flag indicates that the processor was able to correct the event condition. 

OVER 62 Machine check overflow flag indicates that a machine check error occurred while the results of a 
previous error were still in the register bank (i.e., the VAL bit was already set in the 
MC_STATUS register). The processor sets the OVER flag and software is responsible for clearing it. 
Enabled errors are written over disabled errors, and uncorrected errors are written over corrected 
events. Uncorrected errors are not written over previous valid uncorrected errors. 

VAL 63 MC_STATUS register valid flag indicates that the information within the MC_STATUS register is valid. 


When this flag is set, the processor follows the rules given for the OVER flag in the MC_STATUS register 
when overwriting previously valid entries. The processor sets the VAL flag and software is responsible 
for clearing it. 


16.13.1.1 Processor Machine Check Status Register MCA Error Code Definition 


Intel Xeon processor MP 7100 series use compound MCA Error Codes for logging its CBC internal machine check 
errors, L3 Errors, and Bus/Interconnect Errors. It defines additional Machine Check error types 
(IA32_MC4_STATUS[15:0]) beyond those defined in Chapter 15. Table 16-42 lists these model-specific MCA error 
codes. Error code details are specified in MC4_STATUS [31:16] (see Section 16.13.3), the “Model Specific Error 
Code” field. The information in the “Other_Info” field (MC4_STATUS[56:32]) is common to the three processor 
error types and contains a correctable event count and specifies the MC4_MISC register format. 


Table 16-42. Incremental MCA Error Code for Intel Xeon Processor MP 7100 


Processor MCA_Error_Code (MC4_STATUS[15:0]) 


Type | Error Code 


Binary Encoding Meaning 


C Internal Error 0000 0100 0000 0000 | Internal Error Type Code 

A L3 Tag Error 0000 0001 0000 1011 | L3 Tag Error Type Code 

B Bus and 0000 100x 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 
Interconnect implementations 
Error 0000 101x 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 


implementations 


0000 110x 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 
implementations 


0000 1110 0000 1111 | Bus and Interconnection Error Type Code 


0000 1111 0000 1111 | Not used but this encoding is reserved for compatibility with other MCA 
implementations 


The Bold faced binary encodings are the only encodings used by the processor for MC4_STATUS[15:0]. 


16.13.2 Other_Info Field (all MCA Error Types) 


The MC4_STATUS[56:32] field is common to the processor's three MCA error types (A, B & C). 
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Table 16-43. Other Information Field Bit Definition 


Bit Field Name _ | Bits Description 
39:32 8-bit Correctable | Holds a count of the number of correctable events since cold reset. This is a saturating counter; 
Event Count the counter begins at 1 (with the first error) and saturates at a count of 255. 
41:40 MC4_MISC The value in this field specifies the format of information in the MC4_MISC register. Currently, 
format type only two values are defined. Valid only when MISCV is asserted. 
43:42 - Reserved 
51:44 ECC syndrome ECC syndrome value for a correctable ECC event when the “Valid ECC syndrome” bit is asserted 
52 Valid ECC Set when correctable ECC event supplies the ECC syndrome 
syndrome 
54:53 Threshold-Based | 00: No tracking - No hardware status tracking is provided for the structure reporting this event. 
Error Status 01: Green - Status tracking is provided for the structure posting the event; the current status is 
green (below threshold). 
10: Yellow - Status tracking is provided for the structure posting the event; the current status is 
yellow (above threshold). 
11: Reserved for future use 
Valid only if Valid bit (bit 63) is set 
Undefined if the UC bit (bit 61) is set 
56:55 - Reserved 


16.13.3 Processor Model Specific Error Code Field 


16.13.3.1 MCA Error Type A: L3 Error 


Note: The Model Specific Error Code field in MC4_STATUS (bits 31:16). 
Table 16-44. Type A: L3 Error Codes 
Bit Sub-Field | Description Legal Value(s) 
Num Name 
18:16 L3 Error Describes the L3 | O00 - No error 
Code error 001 - More than one way reporting a correctable event 
encountered : 
010 - More than one way reporting an uncorrectable error 
011 - More than one way reporting a tag hit 
100 - No error 
101 - One way reporting a correctable event 
110 - One way reporting an uncorrectable error 
111 - One or more ways reporting a correctable event while one or more ways are 
reporting an uncorrectable error 
20:19 - Reserved 00 
31:21 - Fixed pattern 0010_0000_000 
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16.13.3.2 Processor Model Specific Error Code Field Type B: Bus and Interconnect Error 


Note: The Model Specific Error Code field in MC4_STATUS (bits 31:16). 


Table 16-45. Type B Bus and Interconnect Error Codes 


BitNum_ | Sub-Field Name Description 

16 FSB Request Parity Parity error detected during FSB request phase 

17 CoreO Addr Parity Parity error detected on Core O request's address field 

18 Corel Addr Parity Parity error detected on Core 1 request's address field 

19 Reserved 

20 FSB Response Parity Parity error on FSB response field detected 

21 FSB Data Parity FSB data parity error on inbound data detected 

22 CoreO Data Parity Data parity error on data received from Core O detected 

23 Corel Data Parity Data parity error on data received from Core 1 detected 

24 IDS Parity Detected an Enhanced Defer parity error (phase A or phase B) 
25 FSB Inbound Data ECC | Data ECC event to error on inbound data (correctable or uncorrectable) 
26 FSB Data Glitch Pad logic detected a data strobe ‘glitch’ (or sequencing error) 

27 FSB Address Glitch Pad logic detected a request strobe ‘glitch’ (or sequencing error) 
31:28 “-- Reserved 


Exactly one of the bits defined in the preceding table will be set for a Bus and Interconnect Error. The Data ECC can 
be correctable or uncorrectable (the MC4_STATUS.UC bit, of course, distinguishes between correctable and uncor- 
rectable cases with the Other_Info field possibly providing the ECC Syndrome for correctable errors). All other 
errors for this processor MCA Error Type are uncorrectable. 


16.13.3.3 Processor Model Specific Error Code Field Type C: Cache Bus Controller Error 


Table 16-46. Type C Cache Bus Controller Error Codes 


MC4_STATUS[31:16] (MSCE) Value 


Error Description 


0000_0000_0000_0001 0001H 


Inclusion Error from Core 0 


0000_0000_0000_0010 0002H 


Inclusion Error from Core 1 


0000_0000_0000_0011 0003H 


Write Exclusive Error from Core 0 


0000_0000_0000_0100 0004H 


Write Exclusive Error from Core 1 


0000_0000_0000_0101 0005H 


Inclusion Error from FSB 


0000_0000_0000_0110 OOO6H 


SNP Stall Error from FSB 


0000_0000_0000_0111 0007H 


Write Stall Error from FSB 


0000_0000_0000_1000 0008H 


FSB Arb Timeout Error 


0000_0000_0000_1001 0009H 


CBC OOD Queue Underflow/overflow 


0000_0001_0000_0000 0100H 


Enhanced Intel SpeedStep Technology TM1-TM2 Error 


0000_0010_0000_0000 0200H 
0000_0011_0000_0000 0300H 


Internal Timeout error 


Internal Timeout Error 


0000_0100_0000_0000 0400H 


Intel? Cache Safe Technology Queue Full Error or Disabled-ways-in-a-set overflow 
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Table 16-46. Type C Cache Bus Controller Error Codes (Contd.) 


MC4_STATUS[31:16] (MSCE) Value Error Description 

1100_0000_0000_0001 CO01H Correctable ECC event on outgoing FSB data 
1100_0000_0000_0010 C002H Correctable ECC event on outgoing Core O data 
1100_0000_0000_0100 CO04H Correctable ECC event on outgoing Core 1 data 
1110_0000_0000_0001 €001H Uncorrectable ECC error on outgoing FSB data 
1110_0000_0000_0010 E002H Uncorrectable ECC error on outgoing Core O data 
1110_0000_0000_0100 E004H Uncorrectable ECC error on outgoing Core 1 data 
— all other encodings — Reserved 


All errors - except for the correctable ECC types - in this table are uncorrectable. The correctable ECC events may 
supply the ECC syndrome in the Other_Info field of the MC4_STATUS MSR. 


Table 16-47. Decoding Family OFH Machine Check Codes for Cache Hierarchy Errors 


Type Bit No. | Bit Function Bit Description 

MCA error 15:0 

codes! 

Model 17:16 | Tag Error Code Contains the tag error code for this machine check error: 
specific error 00 = No error detected 

codes 


01 = Parity error on tag miss with a clean line 
10 = Parity error/multiple tag match on tag hit 
11 = Parity error/multiple tag match on tag miss 


19:18 | Data Error Code Contains the data error code for this machine check error: 
00 = No error detected 

01 = Single bit error 

10 = Double bit error on a clean line 

11 = Double bit error on a modified line 


20 L3 Error This bit is set if the machine check error originated in the L3 it can be ignored for 
invalid PIC request errors): 


1 =L3 error 
0 =L2 error 


21 Invalid PIC Request | Indicates error due to invalid PIC request access was made to PIC space with WB 
memory): 


1 = Invalid PIC request error 
O = No invalid PIC request error 


31:22 | Reserved Reserved 
Other 39:32 | 8-bit Error Count Holds a count of the number of errors since reset. The counter begins at O for the 
Information first error and saturates at a count of 255. 
56:40 | Reserved Reserved 
Status 63:57 
register 
validity 
indicators 
NOTES: 


1. These fields are architecturally defined. Refer to Chapter 15, “Machine-Check Architecture,” for more information. 
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CHAPTER 17 
DEBUG, BRANCH PROFILE, TSC, AND INTEL® RESOURCE DIRECTOR 
TECHNOLOGY (INTEL® RDT) FEATURES 


Intel 64 and IA-32 architectures provide debug facilities for use in debugging code and monitoring performance. 
These facilities are valuable for debugging application software, system software, and multitasking operating 
systems. Debug support is accessed using debug registers (DRO through DR7) and model-specific registers 
(MSRs): 


Debug registers hold the addresses of memory and I/O locations called breakpoints. Breakpoints are user- 
selected locations in a program, a data-storage area in memory, or specific I/O ports. They are set where a 
programmer or system designer wishes to halt execution of a program and examine the state of the processor 
by invoking debugger software. A debug exception (#DB) is generated when a memory or I/O access is made 
to a breakpoint address. 


MSRs monitor branches, interrupts, and exceptions; they record addresses of the last branch, interrupt or 
exception taken and the last branch taken before an interrupt or exception. 


Time stamp counter is described in Section 17.17, “Time-Stamp Counter”. 


Features which allow monitoring of shared platform resources such as the L3 cache are described in Section 
17.18, “Intel® Resource Director Technology (Intel® RDT) Monitoring Features”. 


Features which enable control over shared platform resources are described in Section 17.19, “Intel® Resource 
Director Technology (Intel® RDT) Allocation Features”. 


17.1 OVERVIEW OF DEBUG SUPPORT FACILITIES 


The following processor facilities support debugging and performance monitoring: 


Debug exception (#DB) — Transfers program control to a debug procedure or task when a debug event 
occurs. 


Breakpoint exception (#BP) — See breakpoint instruction (INT3) below. 
Breakpoint-address registers (DRO through DR3) — Specifies the addresses of up to 4 breakpoints. 


Debug status register (DR6) — Reports the conditions that were in effect when a debug or breakpoint 
exception was generated. 


Debug control register (DR7) — Specifies the forms of memory or I/O access that cause breakpoints to be 
generated. 


T (trap) flag, TSS — Generates a debug exception (#DB) when an attempt is made to switch to a task with 
the T flag set in its TSS. 


RF (resume) flag, EFLAGS register — Suppresses multiple exceptions to the same instruction. 


TF (trap) flag, EFLAGS register — Generates a debug exception (#DB) after every execution of an 
instruction. 


Breakpoint instruction (INT3) — Generates a breakpoint exception (#BP) that transfers program control to 
the debugger procedure or task. This instruction is an alternative way to set instruction breakpoints. It is 
especially useful when more than four breakpoints are desired, or when breakpoints are being placed in the 
source code. 


Last branch recording facilities — Store branch records in the last branch record (LBR) stack MSRs for the 
most recent taken branches, interrupts, and/or exceptions in MSRs. A branch record consist of a branch-from 
and a branch-to instruction address. Send branch records out on the system bus as branch trace messages 
(BTMs). 


These facilities allow a debugger to be called as a separate task or as a procedure in the context of the current 
program or task. The following conditions can be used to invoke the debugger: 


Task switch to a specific task. 
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¢ Execution of the breakpoint instruction. 

° Execution of any instruction. 

° Execution of an instruction at a specified address. 

® Read or write to a specified memory address/range. 
° Write to a specified memory address/range. 

¢ Input from a specified I/O address/range. 

* Output to a specified I/O address/range. 

¢ Attempt to change the contents of a debug register. 


17.2 DEBUG REGISTERS 


Eight debug registers (see Figure 17-1 for 32-bit operation and Figure 17-2 for 64-bit operation) control the debug 
operation of the processor. These registers can be written to and read using the move to/from debug register form 
of the MOV instruction. A debug register may be the source or destination operand for one of these instructions. 


31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
LEN RW LEN RW LEN RW LEN RW 00 G 0 s 1 Gey ie Ge ey | Se] IE. aS |e Ge | tk DR7 

3 3 2 2 1 1 0 0 D M Ee | iE) Be Be | 22) 4 1 0; 0 

31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 

is B|B/B B)B)/B/B 

@ 1 1 1 1 1 a 1 

Reserved (set to 1) \ tilslo 312111] 0 DR6 

31 0 
DRS 

31 0 
DR4 

31 0 
Breakpoint 3 Linear Address DR3 

31 0 
Breakpoint 2 Linear Address DR2 

31 0 
Breakpoint 1 Linear Address DR1 

31 0 
Breakpoint 0 Linear Address DRO 

Reserved 


Figure 17-1. Debug Registers 
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Debug registers are privileged resources; a MOV instruction that accesses these registers can only be executed in 
real-address mode, in SMM or in protected mode at a CPL of 0. An attempt to read or write the debug registers 
from any other privilege level generates a general-protection exception (#GP). 


The primary function of the debug registers is to set up and monitor from 1 to 4 breakpoints, numbered 0 though 
3. For each breakpoint, the following information can be specified: 


¢ The linear address where the breakpoint is to occur. 

° The length of the breakpoint location: 1, 2, 4, or 8 bytes (refer to the notes in Section 17.2.4). 
¢ The operation that must be performed at the address for a debug exception to be generated. 

¢ Whether the breakpoint is enabled. 

¢ Whether the breakpoint condition was present when the debug exception was generated. 


The following paragraphs describe the functions of flags and fields in the debug registers. 


17.2.1 Debug Address Registers (DRO-DR3) 


Each of the debug-address registers (DRO through DR3) holds the 32-bit linear address of a breakpoint (see 
Figure 17-1). Breakpoint comparisons are made before physical address translation occurs. The contents of debug 
register DR7 further specifies breakpoint conditions. 


17.2.2 Debug Registers DR4 and DR5 


Debug registers DR4 and DRS are reserved when debug extensions are enabled (when the DE flag in control 
register CR4 is set) and attempts to reference the DR4 and DRS registers cause invalid-opcode exceptions (#UD). 
When debug extensions are not enabled (when the DE flag is clear), these registers are aliased to debug registers 
DR6 and DR7. 


17.2.3. Debug Status Register (DR6) 


The debug status register (DR6) reports debug conditions that were sampled at the time the last debug exception 
was generated (see Figure 17-1). Updates to this register only occur when an exception is generated. The flags in 
this register show the following information: 


¢ BO through B3 (breakpoint condition detected) flags (bits 0 through 3) — Indicates (when set) that its 
associated breakpoint condition was met when a debug exception was generated. These flags are set if the 
condition described for each breakpoint by the LENn, and R/Wn flags in debug control register DR7 is true. They 
may or may not be set if the breakpoint is not enabled by the Ln or the Gn flags in register DR7. Therefore on 
a #DB, a debug handler should check only those BO-B3 bits which correspond to an enabled breakpoint. 


¢ BD (debug register access detected) flag (bit 13) — Indicates that the next instruction in the instruction 
stream accesses one of the debug registers (DRO through DR7). This flag is enabled when the GD (general 
detect) flag in debug control register DR7 is set. See Section 17.2.4, “Debug Control Register (DR7),” for 
further explanation of the purpose of this flag. 


° BS (single step) flag (bit 14) — Indicates (when set) that the debug exception was triggered by the single- 
step execution mode (enabled with the TF flag in the EFLAGS register). The single-step mode is the highest- 
priority debug exception. When the BS flag is set, any of the other debug status bits also may be set. 


° BT (task switch) flag (bit 15) — Indicates (when set) that the debug exception resulted from a task switch 
where the T flag (debug trap flag) in the TSS of the target task was set. See Section 7.2.1, “Task-State 
Segment (TSS),” for the format of a TSS. There is no flag in debug control register DR7 to enable or disable this 
exception; the T flag of the TSS is the only enabling flag. 


¢ RTM (restricted transactional memory) flag (bit 16) — Indicates (when clear) that a debug exception 
(#DB) or breakpoint exception (#BP) occurred inside an RTM region while advanced debugging of RTM trans- 
actional regions was enabled (see Section 17.3.3). This bit is set for any other debug exception (including all 
those that occur when advanced debugging of RTM transactional regions is not enabled). This bit is always 1 if 
the processor does not support RTM. 
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Certain debug exceptions may clear bits 0-3. The remaining contents of the DR6 register are never cleared by the 
processor. To avoid confusion in identifying debug exceptions, debug handlers should clear the register (except 
bit 16, which they should set) before returning to the interrupted task. 


17.2.4 Debug Control Register (DR7) 


The debug control register (DR7) enables or disables breakpoints and sets breakpoint conditions (see Figure 17-1). 
The flags and fields in this register control the following things: 


LO through L3 (local breakpoint enable) flags (bits 0, 2, 4, and 6) — Enables (when set) the breakpoint 
condition for the associated breakpoint for the current task. When a breakpoint condition is detected and its 
associated Ln flag is set, a debug exception is generated. The processor automatically clears these flags on 
every task switch to avoid unwanted breakpoint conditions in the new task. 


GO through G3 (global breakpoint enable) flags (bits 1, 3, 5, and 7) — Enables (when set) the 
breakpoint condition for the associated breakpoint for all tasks. When a breakpoint condition is detected and its 
associated Gn flag is set, a debug exception is generated. The processor does not clear these flags on a task 
switch, allowing a breakpoint to be enabled for all tasks. 


LE and GE (local and global exact breakpoint enable) flags (bits 8, 9) — This feature is not supported in 
the P6 family processors, later IA-32 processors, and Intel 64 processors. When set, these flags cause the 
processor to detect the exact instruction that caused a data breakpoint condition. For backward and forward 
compatibility with other Intel processors, we recommend that the LE and GE flags be set to 1 if exact 
breakpoints are required. 


RTM (restricted transactional memory) flag (bit 11) — Enables (when set) advanced debugging of RTM 
transactional regions (see Section 17.3.3). This advanced debugging is enabled only if IA32_DEBUGCTL.RTM is 
also set. 


GD (general detect enable) flag (bit 13) — Enables (when set) debug-register protection, which causes a 
debug exception to be generated prior to any MOV instruction that accesses a debug register. When such a 
condition is detected, the BD flag in debug status register DR6 is set prior to generating the exception. This 
condition is provided to support in-circuit emulators. 


When the emulator needs to access the debug registers, emulator software can set the GD flag to prevent 
interference from the program currently executing on the processor. 


The processor clears the GD flag upon entering to the debug exception handler, to allow the handler access to 
the debug registers. 


R/WO through R/W3 (read/write) fields (bits 16, 17, 20, 21, 24, 25, 28, and 29) — Specifies the 
breakpoint condition for the corresponding breakpoint. The DE (debug extensions) flag in control register CR4 
determines how the bits in the R/Wn fields are interpreted. When the DE flag is set, the processor interprets 
bits as follows: 


00 — Break on instruction execution only. 

01 — Break on data writes only. 

10 — Break on I/O reads or writes. 

11 — Break on data reads or writes but not instruction fetches. 


When the DE flag is clear, the processor interprets the R/Wn bits the same as for the Intel386™ and Intel486™ 
processors, which is as follows: 


00 — Break on instruction execution only. 

01 — Break on data writes only. 

10 — Undefined. 

11 — Break on data reads or writes but not instruction fetches. 


LENO through LENS (Length) fields (bits 18, 19, 22, 23, 26, 27, 30, and 31) — Specify the size of the 
memory location at the address specified in the corresponding breakpoint address register (DRO through DR3). 
These fields are interpreted as follows: 


00 — 1-byte length. 
01 — 2-byte length. 
10 — Undefined (or 8 byte length, see note below). 
11 — 4-byte length. 
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If the corresponding RWn field in register DR7 is 00 (instruction execution), then the LENn field should also be 00. 
The effect of using other lengths is undefined. See Section 17.2.5, “Breakpoint Field Recognition,” below. 


NOTES 


For Pentium® 4 and Intel® Xeon® processors with a CPUID signature corresponding to family 15 
(model 3, 4, and 6), break point conditions permit specifying 8-byte length on data read/write with 
an of encoding 10B in the LENn field. 


Encoding 10B is also supported in processors based on Intel Core microarchitecture or enhanced 
Intel Core microarchitecture, the respective CPUID signatures corresponding to family 6, model 15, 
and family 6, DisplayModel value 23 (see CPUID instruction in Chapter 3, “Instruction Set 
Reference, A-L” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
2A). The Encoding 10B is supported in processors based on Intel® Atom™ microarchitecture, with 
CPUID signature of family 6, DisplayModel value 1CH. The encoding 10B is undefined for other 
processors. 


17.2.5 Breakpoint Field Recognition 


Breakpoint address registers (debug registers DRO through DR3) and the LENn fields for each breakpoint define a 
range of sequential byte addresses for a data or I/O breakpoint. The LENn fields permit specification of a 1-, 2-, 4- 
or 8-byte range, beginning at the linear address specified in the corresponding debug register (DRn). Two-byte 
ranges must be aligned on word boundaries; 4-byte ranges must be aligned on doubleword boundaries, 8-byte 
ranges must be aligned on quadword boundaries. I/O addresses are zero-extended (from 16 to 32 bits, for 
comparison with the breakpoint address in the selected debug register). These requirements are enforced by the 
processor; it uses LENn field bits to mask the lower address bits in the debug registers. Unaligned data or I/O 
breakpoint addresses do not yield valid results. 


A data breakpoint for reading or writing data is triggered if any of the bytes participating in an access is within the 
range defined by a breakpoint address register and its LENn field. Table 17-1 provides an example setup of debug 
registers and data accesses that would subsequently trap or not trap on the breakpoints. 


A data breakpoint for an unaligned operand can be constructed using two breakpoints, where each breakpoint is 
byte-aligned and the two breakpoints together cover the operand. The breakpoints generate exceptions only for 
the operand, not for neighboring bytes. 


Instruction breakpoint addresses must have a length specification of 1 byte (the LENn field is set to 00). Instruction 
breakpoints for other operand sizes are undefined. The processor recognizes an instruction breakpoint address 
only when it points to the first byte of an instruction. If the instruction has prefixes, the breakpoint address must 
point to the first prefix. 
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Table 17-1. Breakpoint Examples 


Debug Register Setup 


Debug Register R/Wn Breakpoint Address LENn 
DRO R/WO = 11 (Read/Write) A0001H LENO = 00 (1 byte) 
DR1 R/W1 = 01 (Write) A0002H LEN1 = 00 (1 byte) 
DR2 R/we = 11 (Read/Write) BOOO02H LEN2 = 01) (2 bytes) 
DR3 R/W3 = 01 (Write) COO00H LEN3 = 11 (4 bytes) 
Data Accesses 
Operation Address Access Length 


(In Bytes) 


Data operations that trap 


- Read or write A0Q001H 1 
- Read or write AQ001H 2 
- Write AQ002H 1 
- Write A0Q002H 2 
- Read or write BOO0O1H 4 
- Read or write BOOO2H 1 
- Read or write BOOO2H 2 
- Write COOO0H 4 
- Write CO0001H 2 
- Write C0003H 1 
Data operations that do not trap 

- Read or write AOOOOH 1 
- Read AQ002H 1 
- Read or write A0Q003H 4 
- Read or write BOOOOH 2 
- Read COOO0OH 2 
- Read or write CO004H 4 


17.2.6 Debug Registers and Intel® 64 Processors 


For Intel 64 architecture processors, debug registers DRO-DR7 are 64 bits. In 16-bit or 32-bit modes (protected 
mode and compatibility mode), writes to a debug register fill the upper 32 bits with zeros. Reads from a debug 
register return the lower 32 bits. In 64-bit mode, MOV DRn instructions read or write all 64 bits. Operand-size 


prefixes are ignored. 


In 64-bit mode, the upper 32 bits of DR6 and DR7 are reserved and must be written with zeros. Writing 1 to any of 
the upper 32 bits results in a #GP(0) exception (see Figure 17-2). All 64 bits of DRO-DR3 are writable by software. 
However, MOV DRn instructions do not check that addresses written to DRO-DR3 are in the linear-address limits of 
the processor implementation (address matching is supported only on valid addresses generated by the processor 


implementation). Break point conditions for 8-byte memory read/writes are supported in all modes. 


17.3. DEBUG EXCEPTIONS 


The Intel 64 and IA-32 architectures dedicate two interrupt vectors to handling debug exceptions: vector 1 (debug 
exception, #DB) and vector 3 (breakpoint exception, #BP). The following sections describe how these exceptions 


are generated and typical exception handler operations. 


17-6 Vol. 3B 


DEBUG, BRANCH PROFILE, TSC, AND INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) FEATURES 


63 32 
DR7 

31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
LEN RW LEN RW LEN RW LEN RW o0 0 |S o o 1/GJ hb} GPL] G]TLI]G]|lLy_G Fe DR7 

3 3 2 2 1 1 0 0 D | BS) Bh | 8. | 2 2) 4 1 0| 0 

63 32 
DR6 

31 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 

R B/;B |B B;/B|B|B 
Reserved (set to 1) i tislo Oa a a) a a a4 1 3/2/11] 0 DR6 

63 32 
DR5 

63 32 
DR4 

63 32 
Breakpoint 3 Linear Address DR3 

63 32 
Breakpoint 2 Linear Address DR2 

63 32 
Breakpoint 1 Linear Address DR1 

63 32 
Breakpoint 0 Linear Address DRO 


Figure 17-2. DR6/DR7 Layout on Processors Supporting Intel® 64 Architecture 


17.3.1 Debug Exception (#DB)—Interrupt Vector 1 


The debug-exception handler is usually a debugger program or part of a larger software system. The processor 
generates a debug exception for any of several conditions. The debugger checks flags in the DR6 and DR7 registers 
to determine which condition caused the exception and which other conditions might apply. Table 17-2 shows the 
states of these flags following the generation of each kind of breakpoint condition. 


Instruction-breakpoint and general-detect condition (see Section 17.3.1.3, “General-Detect Exception Condition”) 
result in faults; other debug-exception conditions result in traps. The debug exception may report one or both at 
one time. The following sections describe each class of debug exception. 
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The INT1 instruction generates a debug exception as a trap. Hardware vendors may use the INT1 instruction for 
hardware debug. For that reason, Intel recommends software vendors instead use the INT3 instruction for software 
breakpoints. 


See also: Chapter 6, “Interrupt 1—Debug Exception (#DB),” in the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 3A. 


Table 17-2. Debug Exception Conditions 


Debug or Breakpoint Condition DR6 Flags Tested DR7 Flags Tested Exception Class 
Single-step trap BS =1 Trap 
Instruction breakpoint, at addresses defined by DRnand | Bn=1 and R/Wn =0 Fault 
LENn (Gn or Ln = 1) 

Data write breakpoint, at addresses defined by DRn and Bn=1 and R/Wn = 1 Trap 
LENn (Gn or Ln = 1) 

1/O read or write breakpoint, at addresses defined by DRn | Bn= 1 and R/Wn = 2 Trap 
and LENn (Gn or Ln = 1) 

Data read or write (but not instruction fetches), at Bn=1 and R/Wn = 3 Trap 
addresses defined by DRn and LENn (Gn or Ln = 1) 

General detect fault, resulting from an attempt to modify | BD = 1 None Fault 
debug registers (usually in conjunction with in-circuit 

emulation) 

Task switch BT =1 None Trap 
INT1 instruction None None Trap 


17.3.1.1.— Instruction-Breakpoint Exception Condition 


The processor reports an instruction breakpoint when it attempts to execute an instruction at an address specified 
in a breakpoint-address register (DRO through DR3) that has been set up to detect instruction execution (R/W flag 
is set to 0). Upon reporting the instruction breakpoint, the processor generates a fault-class, debug exception 
(#DB) before it executes the target instruction for the breakpoint. 


Instruction breakpoints are the highest priority debug exceptions. They are serviced before any other exceptions 
detected during the decoding or execution of an instruction. However, if an instruction breakpoint is placed on an 
instruction located immediately after a POP SS/MOV SS instruction, the breakpoint will be suppressed as if 
EFLAGS.RF were 1 (see the next paragraph and Section 6.8.3, “Masking Exceptions and Interrupts When 
Switching Stacks,” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). 


Because the debug exception for an instruction breakpoint is generated before the instruction is executed, if the 
instruction breakpoint is not removed by the exception handler; the processor will detect the instruction breakpoint 
again when the instruction is restarted and generate another debug exception. To prevent looping on an instruction 
breakpoint, the Intel 64 and IA-32 architectures provide the RF flag (resume flag) in the EFLAGS register (see 
Section 2.3, “System Flags and Fields in the EFLAGS Register,” in the Intel® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 3A). When the RF flag is set, the processor ignores instruction breakpoints. 


All Intel 64 and IA-32 processors manage the RF flag as follows. The RF Flag is cleared at the start of the instruction 
after the check for instruction breakpoints, CS limit violations, and FP exceptions. Task Switches and IRETD/IRETQ 
instructions transfer the RF image from the TSS/stack to the EFLAGS register. 


When calling an event handler, Intel 64 and IA-32 processors establish the value of the RF flag in the EFLAGS image 
pushed on the stack: 


° For any fault-class exception except a debug exception generated in response to an instruction breakpoint, the 
value pushed for RF is 1. 


° For any interrupt arriving after any iteration of a repeated string instruction but the last iteration, the value 
pushed for RF is 1. 
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¢ For any trap-class exception generated by any iteration of a repeated string instruction but the last iteration, 
the value pushed for RF is 1. 


¢ For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was 
called. This includes: 


— Debug exceptions generated in response to instruction breakpoints 


— Hardware-generated interrupts arriving between instructions (including those arriving after the last 
iteration of a repeated string instruction) 


— Trap-class exceptions generated after an instruction completes (including those generated after the last 
iteration of a repeated string instruction) 


— Software-generated interrupts (RF is pushed as 0, since it was cleared at the start of the software interrupt) 


As noted above, the processor does not set the RF flag prior to calling the debug exception handler for debug 
exceptions resulting from instruction breakpoints. The debug exception handler can prevent recurrence of the 
instruction breakpoint by setting the RF flag in the EFLAGS image on the stack. If the RF flag in the EFLAGS image 
is set when the processor returns from the exception handler, it is copied into the RF flag in the EFLAGS register by 
IRETD/IRETQ or a task switch that causes the return. The processor then ignores instruction breakpoints for the 
duration of the next instruction. (Note that the POPF, POPFD, and IRET instructions do not transfer the RF image 
into the EFLAGS register.) Setting the RF flag does not prevent other types of debug-exception conditions (such as, 
I/O or data breakpoints) from being detected, nor does it prevent non-debug exceptions from being generated. 


For the Pentium processor, when an instruction breakpoint coincides with another fault-type exception (such as a 
page fault), the processor may generate one spurious debug exception after the second exception has been 
handled, even though the debug exception handler set the RF flag in the EFLAGS image. To prevent a spurious 
exception with Pentium processors, all fault-class exception handlers should set the RF flag in the EFLAGS image. 


17.3.1.2 Data Memory and I/O Breakpoint Exception Conditions 


Data memory and I/O breakpoints are reported when the processor attempts to access a memory or I/O address 
specified in a breakpoint-address register (DRO through DR3) that has been set up to detect data or I/O accesses 
(R/W flag is set to 1, 2, or 3). The processor generates the exception after it executes the instruction that made the 
access, so these breakpoint condition causes a trap-class exception to be generated. 


Because data breakpoints are traps, an instruction that writes memory overwrites the original data before the 
debug exception generated by a data breakpoint is generated. If a debugger needs to save the contents of a write 
breakpoint location, it should save the original contents before setting the breakpoint. The handler can report the 
saved value after the breakpoint is triggered. The address in the debug registers can be used to locate the new 
value stored by the instruction that triggered the breakpoint. 


If a data breakpoint is detected during an iteration of a string instruction executed with fast-string operation (see 
Section 7.3.9.3 of Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1), delivery of the 
resulting debug exception may be delayed until completion of the corresponding group of iterations. 


Intel486 and later processors ignore the GE and LE flags in DR7. In Intel386 processors, exact data breakpoint 
matching does not occur unless it is enabled by setting the LE and/or the GE flags. 


For repeated INS and OUTS instructions that generate an I/O-breakpoint debug exception, the processor gener- 
ates the exception after the completion of the first iteration. Repeated INS and OUTS instructions generate a data- 
breakpoint debug exception after the iteration in which the memory address breakpoint location is accessed. 


If an execution of the MOV or POP instruction loads the SS register and encounters a data breakpoint, the resulting 
debug exception is delivered after completion of the next instruction (the one after the MOV or POP). 


Any pending data or I/O breakpoints are lost upon delivery of an exception. For example, if a machine-check 
exception (#MC) occurs following an instruction that encounters a data breakpoint (but before the resulting debug 
exception is delivered), the data breakpoint is lost. If a MOV or POP instruction that loads the SS register encoun- 
ters a data breakpoint, the data breakpoint is lost if the next instruction causes a fault. 


Delivery of events due to INT n, INT3, or INTO does not cause a loss of data breakpoints. If a MOV or POP instruc- 
tion that loads the SS register encounters a data breakpoint, and the next instruction is software interrupt (INT n, 
INT3, or INTO), a debug exception (#DB) resulting from a data breakpoint will be delivered after the transition to 
the software-interrupt handler. The #DB handler should account for the fact that the #DB may have been delivered 
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after a invocation of a software-interrupt handler, and in particular that the CPL may have changed between recog- 
nition of the data breakpoint and delivery of the #DB. 


17.3.1.3 General-Detect Exception Condition 


When the GD flag in DR7 is set, the general-detect debug exception occurs when a program attempts to access any 
of the debug registers (DRO through DR7) at the same time they are being used by another application, such as an 
emulator or debugger. This protection feature guarantees full control over the debug registers when required. The 
debug exception handler can detect this condition by checking the state of the BD flag in the DR6 register. The 
processor generates the exception before it executes the MOV instruction that accesses a debug register, which 
causes a fault-class exception to be generated. 


17.3.1.4 Single-Step Exception Condition 


The processor generates a single-step debug exception if (while an instruction is being executed) it detects that the 
TF flag in the EFLAGS register is set. The exception is a trap-class exception, because the exception is generated 

after the instruction is executed. The processor will not generate this exception after the instruction that sets the 
TF flag. For example, if the POPF instruction is used to set the TF flag, a single-step trap does not occur until after 
the instruction that follows the POPF instruction. 


The processor clears the TF flag before calling the exception handler. If the TF flag was set in a TSS at the time of 
a task switch, the exception occurs after the first instruction is executed in the new task. 


The TF flag normally is not cleared by privilege changes inside a task. The INT n, INT3, and INTO instructions, 
however, do clear this flag. Therefore, software debuggers that single-step code must recognize and emulate INT n 
or INTO instructions rather than executing them directly. To maintain protection, the operating system should 
check the CPL after any single-step trap to see if single stepping should continue at the current privilege level. 


The interrupt priorities guarantee that, if an external interrupt occurs, single stepping stops. When both an external 
interrupt and a single-step interrupt occur together, the single-step interrupt is processed first. This operation 
clears the TF flag. After saving the return address or switching tasks, the external interrupt input is examined 
before the first instruction of the single-step handler executes. If the external interrupt is still pending, then it is 
serviced. The external interrupt handler does not run in single-step mode. To single step an interrupt handler, 
single step an INT r instruction that calls the interrupt handler. 


If an occurrence of the MOV or POP instruction loads the SS register executes with EFLAGS.TF = 1, no single-step 
debug exception occurs following the MOV or POP instruction. 


17.3.1.5 Task-Switch Exception Condition 


The processor generates a debug exception after a task switch if the T flag of the new task's TSS is set. This excep- 
tion is generated after program control has passed to the new task, and prior to the execution of the first instruc- 
tion of that task. The exception handler can detect this condition by examining the BT flag of the DR6 register. 


If entry 1 (#DB) in the IDT is a task gate, the T bit of the corresponding TSS should not be set. Failure to observe 
this rule will put the processor in a loop. 


17.3.2 Breakpoint Exception (HBP)—Interrupt Vector 3 


The breakpoint exception (interrupt 3) is caused by execution of an INT3 instruction. See Chapter 6, 

“Interrupt 3—Breakpoint Exception (#BP).” Debuggers use breakpoint exceptions in the same way that they use 
the breakpoint registers; that is, as a mechanism for suspending program execution to examine registers and 
memory locations. With earlier IA-32 processors, breakpoint exceptions are used extensively for setting instruction 
breakpoints. 


With the Intel386 and later IA-32 processors, it is more convenient to set breakpoints with the breakpoint-address 
registers (DRO through DR3). However, the breakpoint exception still is useful for breakpointing debuggers, 
because a breakpoint exception can call a separate exception handler. The breakpoint exception is also useful when 
it is necessary to set more breakpoints than there are debug registers or when breakpoints are being placed in the 
source code of a program under development. 
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17.3.3. Debug Exceptions, Breakpoint Exceptions, and Restricted Transactional Memory 
(RTM) 


Chapter 16, “Programming with Intel® Transactional Synchronization Extensions,” of Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 1 describes Restricted Transactional Memory (RTM). This is an 
instruction-set interface that allows software to identify transactional regions (or critical sections) using the 
XBEGIN and XEND instructions. 


Execution of an RTM transactional region begins with an XBEGIN instruction. If execution of the region successfully 
reaches an XEND instruction, the processor ensures that all memory operations performed within the region 
appear to have occurred instantaneously when viewed from other logical processors. Execution of an RTM transac- 
tion region does not succeed if the processor cannot commit the updates atomically. When this happens, the 
processor rolls back the execution, a process referred to as a transactional abort. In this case, the processor 
discards all updates performed in the region, restores architectural state to appear as if the execution had not 
occurred, and resumes execution at a fallback instruction address that was specified with the XBEGIN instruction. 


If debug exception (#DB) or breakpoint exception (#BP) occurs within an RTM transaction region, a transactional 
abort occurs, the processor sets EAX[4], and no exception is delivered. 


Software can enable advanced debugging of RTM transactional regions by setting DR7.RTM[bit 11] and 
IA32_DEBUGCTL.RTM[bit 15]. If these bits are both set, the transactional abort caused by a #DB or #BP within an 
RTM transaction region does not resume execution at the fallback instruction address specified with the XBEGIN 
instruction that begin the region. Instead, execution is resumed at that XBEGIN instruction, and a #DB is deliv- 
ered. (A #DB is delivered even if the transactional abort was caused by a #BP.) Such a #DB will clear 
DR6.RTM[bit 16] (all other debug exceptions set DR6[16]). 


17.4 LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING OVERVIEW 


P6 family processors introduced the ability to set breakpoints on taken branches, interrupts, and exceptions, and 
to single-step from one branch to the next. This capability has been modified and extended in the Pentium 4, Intel 
Xeon, Pentium M, Intel® Core™ Solo, Intel® Core™ Duo, Intel® Core™2 Duo, Intel® Core™ i7 and Intel® Atom™ 
processors to allow logging of branch trace messages in a branch trace store (BTS) buffer in memory. 


See the following sections for processor specific implementation of last branch, interrupt and exception recording: 


— Section 17.5, “Last Branch, Interrupt, and Exception Recording (Intel® Core™ 2 Duo and Intel® Atom™ 
Processors)” 


— Section 17.6, “Last Branch, Call Stack, Interrupt, and Exception Recording for Processors based on 
Goldmont Microarchitecture” 


— Section 17.9, “Last Branch, Interrupt, and Exception Recording for Processors based on Intel® Microarchi- 
tecture code name Nehalem” 


— Section 17.10, “Last Branch, Interrupt, and Exception Recording for Processors based on Intel® Microar- 
chitecture code name Sandy Bridge” 


— Section 17.11, “Last Branch, Call Stack, Interrupt, and Exception Recording for Processors based on 
Haswell Microarchitecture” 


— Section 17.12, “Last Branch, Call Stack, Interrupt, and Exception Recording for Processors based on 
Skylake Microarchitecture” 


— Section 17.14, “Last Branch, Interrupt, and Exception Recording (Intel® Core™ Solo and Intel® Core™ 
Duo Processors)” 


— Section 17.15, “Last Branch, Interrupt, and Exception Recording (Pentium M Processors)” 
— Section 17.16, “Last Branch, Interrupt, and Exception Recording (P6 Family Processors)” 


The following subsections of Section 17.4 describe common features of profiling branches. These features are 
generally enabled using the IA32_DEBUGCTL MSR (older processor may have implemented a subset or model- 
specific features, see definitions of MSR_DEBUGCTLA, MSR_DEBUGCTLB, MSR_DEBUGCTL). 
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17.4.1. 1IA32_DEBUGCTL MSR 


The IA32_DEBUGCTL MSR provides bit field controls to enable debug trace interrupts, debug trace stores, trace 
messages enable, single stepping on branches, last branch record recording, and to control freezing of LBR stack 
or performance counters on a PMI request. IA32_DEBUGCTL MSR is located at register address 01D9H. 


See Figure 17-3 for the MSR layout and the bullets below for a description of the flags: 


LBR (last branch/interrupt/ exception) flag (bit 0) — When set, the processor records a running trace of 
the most recent branches, interrupts, and/or exceptions taken by the processor (prior to a debug exception 
being generated) in the last branch record (LBR) stack. For more information, see the Section 17.5.1, “LBR 
Stack” (Intel® Core™2 Duo and Intel® Atom™ Processor Family) and Section 17.9.1, “LBR Stack” (processors 
based on Intel® Microarchitecture code name Nehalem). 


BTF (single-step on branches) flag (bit 1) — When set, the processor treats the TF flag in the EFLAGS 
register as a “single-step on branches” flag rather than a “single-step on instructions” flag. This mechanism 
allows single-stepping the processor on taken branches. See Section 17.4.3, “Single-Stepping on Branches,” 
for more information about the BTF flag. 


TR (trace message enable) flag (bit 6) — When set, branch trace messages are enabled. When the 
processor detects a taken branch, interrupt, or exception; it sends the branch record out on the system bus as 
a branch trace message (BTM). See Section 17.4.4, “Branch Trace Messages,” for more information about the 
TR flag. 


BTS (branch trace store) flag (bit 7) — When set, the flag enables BTS facilities to log BTMs to a memory- 
resident BTS buffer that is part of the DS save area. See Section 17.4.9, “BTS and DS Save Area.” 


BTINT (branch trace interrupt) flag (bit 8) — When set, the BTS facilities generate an interrupt when the 
BTS buffer is full. When clear, BTMs are logged to the BTS buffer in a circular fashion. See Section 17.4.5, “Branch 
Trace Store (BTS),” for a description of this mechanism. 


31 1514 121110987654321 0 


Reserved 


RTM 
FREEZE_WHILE_SMM 
FREEZE_PERFMON_ON_PMI 
FREEZE_LBRS_ON_PMI 
BTS_OFF_USR — BTS off in user code 
BTS_OFF_OS — BTS off in OS 
BTINT — Branch trace interrupt 
BTS — Branch trace store 
TR — Trace messages enable 
Reserved 
BTF — Single-step on branches 
LBR — Last branch/interrupt/exception 


Figure 17-3. 1A32_DEBUGCTL MSR for Processors based 
on Intel Core microarchitecture 


BTS_OFF_OS (branch trace off in privileged code) flag (bit 9) — When set, BTS or BTM is skipped if CPL 
is 0. See Section 17.13.2. 


BTS_OFF_USR (branch trace off in user code) flag (bit 10) — When set, BTS or BTM is skipped if CPL is 
greater than 0. See Section 17.13.2. 


FREEZE_LBRS_ON_PMI flag (bit 11) — When set, the LBR stack is frozen on a hardware PMI request (e.g. 
when a counter overflows and is configured to trigger PMI). See Section 17.4.7 for details. 


FREEZE_PERFMON_ON_PMI flag (bit 12) — When set, the performance counters (IA32_PMCx and 
IA32_FIXED_CTRx) are frozen on a PMI request. See Section 17.4.7 for details. 


FREEZE_WHILE_SMM (bit 14) — If this bit is set, upon the delivery of an SMI, the processor will clear all the 
enable bits of IA32_PERF_GLOBAL_CTRL, save a copy of the content of IA32_DEBUGCTL and disable LBR, BTF, 
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TR, and BTS fields of IA32_DEBUGCTL before transferring control to the SMI handler. Subsequently, the enable 
bits of IA32_PERF_GLOBAL_CTRL will be set to 1, the saved copy of IA32_DEBUGCTL prior to SMI delivery will 
be restored, after the SMI handler issues RSM to complete its service. Note that system software must check if 
the processor supports the IA32_DEBUGCTL.FREEZE_WHILE_SMM control bit. 
IA32_DEBUGCTL.FREEZE_WHILE_SMM is supported if IA32_PERF_CAPABILITIES.FREEZE_WHILE_SMM[Bit 
12] is reporting 1. See Section 18.8 for details of detecting the presence of IA32_PERF_CAPABILITIES MSR. 


¢ RTM (bit 15) — If this bit is set, advanced debugging of RTM transactional regions is enabled if DR7.RTM is 
also set. See Section 17.3.3. 


17.4.2 | Monitoring Branches, Exceptions, and Interrupts 


When the LBR flag (bit 0) in the IA32_DEBUGCTL MSR is set, the processor automatically begins recording branch 
records for taken branches, interrupts, and exceptions (except for debug exceptions) in the LBR stack MSRs. 


When the processor generates a debug exception (#DB), it automatically clears the LBR flag before executing the 
exception handler. This action does not clear previously stored LBR stack MSRs. 


A debugger can use the linear addresses in the LBR stack to re-set breakpoints in the breakpoint address registers 
(DRO through DR3). This allows a backward trace from the manifestation of a particular bug toward its source. 


On some processors, if the LBR flag is cleared and TR flag in the IA32_DEBUGCTL MSR remains set, the processor 
will continue to update LBR stack MSRs. This is because those processors use the entries in the LBR stack in the 
process of generating BTM/BTS records. A #DB does not automatically clear the TR flag. 


17.4.3 Single-Stepping on Branches 


When software sets both the BTF flag (bit 1) in the IA32_DEBUGCTL MSR and the TF flag in the EFLAGS register, 
the processor generates a single-step debug exception only after instructions that cause a branch.! This mecha- 
nism allows a debugger to single-step on control transfers caused by branches. This “branch single stepping” helps 
isolate a bug to a particular block of code before instruction single-stepping further narrows the search. The 
processor clears the BTF flag when it generates a debug exception. The debugger must set the BTF flag before 
resuming program execution to continue single-stepping on branches. 


17.4.4 Branch Trace Messages 


Setting the TR flag (bit 6) in the IA32_DEBUGCTL MSR enables branch trace messages (BTMs). Thereafter, when 
the processor detects a branch, exception, or interrupt, it sends a branch record out on the system bus as a BIM. 
A debugging device that is monitoring the system bus can read these messages and synchronize operations with 
taken branch, interrupt, and exception events. 


When interrupts or exceptions occur in conjunction with a taken branch, additional BTMs are sent out on the bus, 
as described in Section 17.4.2, “Monitoring Branches, Exceptions, and Interrupts.” 


For P6 processor family, Pentium M processor family, processors based on Intel Core microarchitecture, TR and LBR 
bits can not be set at the same time due to hardware limitation. The content of LBR stack is undefined when TR is 
set. 


For processors with Intel NetBurst microarchitecture, Intel Atom processors, and Intel Core and related Intel Xeon 
processors both starting with the Nehalem microarchitecture, the processor can collect branch records in the LBR 
stack and at the same time send/store BTMs when both the TR and LBR flags are set in the IA32_DEBUGCTL MSR 
(or the equivalent MSR_DEBUGCTLA, MSR_DEBUGCTLB). 


The following exception applies: 


¢ BTM may not be observable on Intel Atom processor families that do not provide an externally visible system 
bus (i.e., processors based on the Silvermont microarchitecture or later). 


1. Executions of CALL, IRET, and JMP that cause task switches never cause single-step debug exceptions (regardless of the value of the 
BTF flag). A debugger desiring debug exceptions on switches to a task should set the T flag (debug trap flag) in the TSS of that task. 
See Section 7.2.1, “Task-State Segment (TSS).” 
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17.4.4.1. Branch Trace Message Visibility 


Branch trace message (BIM) visibility is implementation specific and limited to systems with a front side bus 
(FSB). BTMs may not be visible to newer system link interfaces or a system bus that deviates from a traditional 
FSB. 


17.4.5 Branch Trace Store (BTS) 


A trace of taken branches, interrupts, and exceptions is useful for debugging code by providing a method of deter- 
mining the decision path taken to reach a particular code location. The LBR flag (bit 0) of IA32_DEBUGCTL provides 
a mechanism for capturing records of taken branches, interrupts, and exceptions and saving them in the last 
branch record (LBR) stack MSRs, setting the TR flag for sending them out onto the system bus as BTMs. The branch 
trace store (BTS) mechanism provides the additional capability of saving the branch records in a memory-resident 
BTS buffer, which is part of the DS save area. The BTS buffer can be configured to be circular so that the most 
recent branch records are always available or it can be configured to generate an interrupt when the buffer is nearly 
full so that all the branch records can be saved. The BTINT flag (bit 8) can be used to enable the generation of inter- 
rupt when the BTS buffer is full. See Section 17.4.9.2, “Setting Up the DS Save Area.” for additional details. 


Setting this flag (BTS) alone can greatly reduce the performance of the processor. CPL-qualified branch trace 
storing mechanism can help mitigate the performance impact of sending/logging branch trace messages. 


17.4.6 CPL-Qualified Branch Trace Mechanism 


CPL-qualified branch trace mechanism is available to a subset of Intel 64 and IA-32 processors that support the 
branch trace storing mechanism. The processor supports the CPL-qualified branch trace mechanism if 
CPUID.O1H:ECX[bit 4] = 1. 


The CPL-qualified branch trace mechanism is described in Section 17.4.9.4. System software can selectively specify 
CPL qualification to not send/store Branch Trace Messages associated with a specified privilege level. Two bit fields, 
BTS_OFF_USR (bit 10) and BTS_OFF_OS (bit 9), are provided in the debug control register to specify the CPL of 
BTMs that will not be logged in the BTS buffer or sent on the bus. 


17.4.7. Freezing LBR and Performance Counters on PMI 


Many issues may generate a performance monitoring interrupt (PMI); a PMI service handler will need to determine 
cause to handle the situation. Two capabilities that allow a PMI service routine to improve branch tracing and 
performance monitoring are available for processors supporting architectural performance monitoring version 2 or 
greater (i.e. CPUID.OAH:EAX[7:0] > 1). These capabilities provides the following interface in IA32_DEBUGCTL to 
reduce runtime overhead of PMI servicing, profiler-contributed skew effects on analysis or counter metrics: 


° Freezing LBRs on PMI (bit 11)— Allows the PMI service routine to ensure the content in the LBR stack are 
associated with the target workload and not polluted by the branch flows of handling the PMI. Depending on the 
version ID enumerated by CPUID.OAH:EAX.ArchPerfMonVerID[bits 7:0], two flavors are supported: 


— Legacy Freeze_LBR_on_PMI is supported for ArchPerfMonVerID <= 3 and ArchPerfMonVerID >1. If 
IA32_DEBUGCTL.Freeze_LBR_On_PMI = 1, the LBR is frozen on the overflowed condition of the buffer 
area, the processor clears the LBR bit (bit 0) in IA32_DEBUGCTL. Software must then re-enable 
IA32_DEBUGCTL.LBR to resume recording branches. When using this feature, software should be careful 
about writes to IA32_DEBUGCTL to avoid re-enabling LBRs by accident if they were just disabled. 


— Streamlined Freeze_LBR_on_PMI is supported for ArchPerfMonVerID >= 4. If 
IA32_DEBUGCTL.Freeze_LBR_On_PMI = 1, the processor behaves as follows: 


¢ sets IA32_PERF_GLOBAL_STATUS.LBR_Frz =1 to disable recording, but does not change the LBR bit 
(bit 0) in IA32_DEBUGCTL. The LBRs are frozen on the overflowed condition of the buffer area. 


¢ Freezing PMCs on PMI (bit 12) — Allows the PMI service routine to ensure the content in the performance 
counters are associated with the target workload and not polluted by the PMI and activities within the PMI 
service routine. Depending on the version ID enumerated by CPUID.OAH:EAX.ArchPerfMonVerlID [bits 7:0], two 
flavors are supported: 
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— Legacy Freeze_Perfmon_on_PMI is supported for ArchPerfMonVerID <= 3 and ArchPerfMonVerID >1. If 
IA32_DEBUGCTL.Freeze_Perfmon_On_PMI = 1, the performance counters are frozen on the counter 
overflowed condition when the processor clears the IA32_PERF_GLOBAL_CTRL MSR (see Figure 18-3). The 
PMCs affected include both general-purpose counters and fixed-function counters (see Section 18.6.2.1, 
“Fixed-function Performance Counters”). Software must re-enable counts by writing is to the corre- 
sponding enable bits in IA32_PERF_GLOBAL_CTRL before leaving a PMI service routine to continue counter 
operation. 

— Streamlined Freeze_Perfmon_on_PMI is supported for ArchPerfMonVerID >= 4. The processor behaves as 
follows: 
¢ sets IA32_PERF_GLOBAL_STATUS.CTR_Frz =1 to disable counting on a counter overflow condition, but 

does not change the IA32_PERF_GLOBAL_CTRL MSR. 


Freezing LBRs and PMCs on PMIs (both legacy and streamlined operation) occur when one of the following applies: 
¢ A performance counter had an overflow and was programmed to signal a PMI in case of an overflow. 


— For the general-purpose counters; enabling PMI is done by setting bit 20 of the IA32_PERFEVTSELx 
register. 


— For the fixed-function counters; enabling PMI is done by setting the 3rd bit in the corresponding 4-bit 
control field of the MSR_PERF_FIXED_CTR_CTRL register (see Figure 18-1) or IA32_FIXED_CTR_CTRL MSR 
(see Figure 18-2). 


¢ The PEBS buffer is almost full and reaches the interrupt threshold. 
¢ The BTS buffer is almost full and reaches the interrupt threshold. 


Table 17-3 compares the interaction of the processor with the PMI handler using the legacy versus streamlined 
Freeza_Perfmon_On_PMI interface. 


Table 17-3. Legacy and Streamlined Operation with Freeze_Perfmon_On_PMI = 1, Counter Overflowed 


Legacy Freeze_Perfmon_On_PMI Streamlined Freeze_Perfmon_On_PMI Comment 


Processor freezes the counters on overflow Processor freezes the counters on overflow Unchanged 


Processor clears [A32_PERF_GLOBAL_CTRL Processor set 
IA32_PERF_GLOBAL_STATUS.CTR_FTZ 


Handler reads |[A32_PERF_GLOBAL_STATUS mask = RDMSR(0x38E) Similar 

(0x38E) to examine which counter(s) overflowed 

Handler services the PMI Handler services the PMI Unchanged 

Handler writes 1s to Handler writes mask into 

IA32_PERF_GLOBAL_OVF_CTL (0x390) IA32_PERF_GLOBAL_OVF_RESET (0x390) 

Processor clears IA32_PERF_GLOBAL_STATUS _ | Processor clears IA32_PERF_GLOBAL_STATUS | Unchanged 

Handler re-enables I[A32_PERF_GLOBAL_CTRL_ |None Reduced software overhead 
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17.4.8 LBR Stack 


The last branch record stack and top-of-stack (TOS) pointer MSRs are supported across Intel 64 and IA-32 
processor families. However, the number of MSRs in the LBR stack and the valid range of TOS pointer value can 
vary between different processor families. Table 17-4 lists the LBR stack size and TOS pointer range for several 
processor families according to the CPUID signatures of DisplayFamily_DisplayModel encoding (see CPUID instruc- 
tion in Chapter 3 of Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A). 


Table 17-4. LBR Stack Size and TOS Pointer Range 


DisplayFamily_DisplayModel Size of LBR Stack |Component of an LBR Entry | Range of TOS Pointer 
06_5CH, 06_5FH 32 FROM_IP, TO_IP 0 to 31 
06_4E€H, 06_5EH, 06_8EH, 06_9EH, 06_55H, 32 FROM_IP, TO_IP, LBR_INFO! 0 to 31 


06_66H, 06_7AH, 06_67H, 06_6AH, 06_6CH, 
06_7DH, 06_7€H, 06_8CH, 06_8DH, 06_6AH, 
06_6CH 


06_3DH, 06_47H, 06_4FH, 06_56H, 06_3CH, 16 FROM_IP, TO_IP Oto15 
06_45H, 06_46H, 06_3FH, 06_2AH, 06_2DH, 
06_3AH, 06_3€H, 06_1AH, 06_1€H, 06_1FH, 
06_2EH, 06_25H, 06_2CH, 06_2FH 


06_17H, 06_1DH, 06_OFH 4 FROM_IP, TO_IP 0 to3 
06_37H, 06_4AH, 06_4CH, 06_4DH, 06_5AH, 8 FROM_IP, TO_IP Oto7 
06_5DH, 06_1CH, 06_26H, 06_27H, 06_35H, 

06_36H 

NOTES: 


1. See Section 17.12. 


The last branch recording mechanism tracks not only branch instructions (like JMP, Jcc, LOOP and CALL instruc- 
tions), but also other operations that cause a change in the instruction pointer (like external interrupts, traps and 
faults). The branch recording mechanisms generally employs a set of MSRs, referred to as last branch record (LBR) 
stack. The size and exact locations of the LBR stack are generally model-specific (see Chapter 2, “Model-Specific 
Registers (MSRs)” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4 for model- 
specific MSR addresses). 


¢ Last Branch Record (LBR) Stack — The LBR consists of N pairs of MSRs (N is listed in the LBR stack size 
column of Table 17-4) that store source and destination address of recent branches (see Figure 17-3): 


— MSR_LASTBRANCH_0O_FROM_IP (address is model specific) through the next consecutive (N-1) MSR 
address store source addresses. 


— MSR_LASTBRANCH_0_TO_IP (address is model specific ) through the next consecutive (N-1) MSR address 
store destination addresses. 


¢ Last Branch Record Top-of-Stack (TOS) Pointer — The lowest significant M bits of the TOS Pointer MSR 
(MSR_LASTBRANCH_TOS, address is model specific) contains an M-bit pointer to the MSR in the LBR stack that 
contains the most recent branch, interrupt, or exception recorded. The valid range of the M-bit POS pointer is 
given in Table 17-4. 


17.4.8.1 _LBR Stack and Intel® 64 Processors 


LBR MSRs are 64-bits. In 64-bit mode, last branch records store the full address. Outside of 64-bit mode, the upper 
32-bits of branch addresses will be stored as 0. 
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MSR_LASTBRANCH_0_FROM_IP through MSR_LASTBRANCH_(N-1)_ FROM_IP 
63 0 


Source Address 


MSR_LASTBRANCH_0_TO_IP through MSR_LASTBRANCH_(N-1)_TO_IP 
63 0 


Destination Address 


Figure 17-4. 64-bit Address Layout of LBR MSR 


Software should query an architectural MSR IA32_PERF_CAPABILITIES[5:0] about the format of the address that 
is stored in the LBR stack. Four formats are defined by the following encoding: 


— 0OO0O000B (32-bit record format) — Stores 32-bit offset in current CS of respective source/destination, 
— 000001B (64-bit LIP record format) — Stores 64-bit linear address of respective source/destination, 


— 000010B (64-bit EIP record format) — Stores 64-bit offset (effective address) of respective 
source/destination. 


— 000011B (64-bit EIP record format) and Flags — Stores 64-bit offset (effective address) of respective 
source/destination. Misprediction info is reported in the upper bit of '"FROM' registers in the LBR stack. See 
LBR stack details below for flag support and definition. 


— 000100B (64-bit EIP record format), Flags and TSX — Stores 64-bit offset (effective address) of 
respective source/destination. Misprediction and TSX info are reported in the upper bits of ‘FROM’ registers 
in the LBR stack. 


— 000101B (64-bit EIP record format), Flags, TSX, LBR_INFO — Stores 64-bit offset (effective 
address) of respective source/destination. Misprediction, TSX, and elapsed cycles since the last LBR update 
are reported in the LBR_INFO MSR stack. 


— 000110B (64-bit LIP record format), Flags, Cycles — Stores 64-bit linear address (CS.Base + 
effective address) of respective source/destination. Misprediction info is reported in the upper bits of 
"FROM' registers in the LBR stack. Elapsed cycles since the last LBR update are reported in the upper 16 bits 
of the 'TO' registers in the LBR stack (see Section 17.6). 


— 000111B (64-bit LIP record format), Flags, LBR_INFO — Stores 64-bit linear address (CS.Base + 
effective address) of respective source/destination. Misprediction, and elapsed cycles since the last LBR 
update are reported in the LBR_INFO MSR stack. 


Processor’s support for the architectural MSR IA32_PERF_CAPABILITIES is provided by 
CPUID.01H:ECX[PERF_CAPAB_MSR] (bit 15). 


17.4.8.2 _LBR Stack and IA-32 Processors 


The LBR MSRs in IA-32 processors introduced prior to Intel 64 architecture store the 32-bit “To Linear Address” and 
“From Linear Address* using the high and low half of each 64-bit MSR. 


17.4.8.3. Last Exception Records and Intel 64 Architecture 


Intel 64 and IA-32 processors also provide MSRs that store the branch record for the last branch taken prior to an 
exception or an interrupt. The location of the last exception record (LER) MSRs are model specific. The MSRs that 
store last exception records are 64-bits. If IA-32e mode is disabled, only the lower 32-bits of the address is 

recorded. If IA-32e mode is enabled, the processor writes 64-bit values into the MSR. In 64-bit mode, last excep- 
tion records store 64-bit addresses; in compatibility mode, the upper 32-bits of last exception records are cleared. 
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17.4.9 BTS and DS Save Area 


The Debug store (DS) feature flag (bit 21), returned by CPUID.1:EDX[21] indicates that the processor provides 
the debug store (DS) mechanism. The DS mechanism allows: 


° BTMs to be stored in a memory-resident BTS buffer. See Section 17.4.5, “Branch Trace Store (BTS).” 


° Processor event-based sampling (PEBS) also uses the DS save area provided by debug store mechanism. The 
capability of PEBS varies across different microarchitectures. See Section 18.6.2.4, “Processor Event Based 
Sampling (PEBS),” and the relevant PEBS sub-sections across the core PMU sections in Chapter 18, “Perfor- 
mance Monitoring.” 


When CPUID.1:EDX[21] is set: 


¢ The BTS_UNAVAILABLE and PEBS_UNAVAILABLE flags in the IA32_MISC_ENABLE MSR indicate (when clear) 
the availability of the BTS and PEBS facilities, including the ability to set the BTS and BTINT bits in the 
appropriate DEBUGCTL MSR. 


¢ The IA32_DS_AREA MSR exists and points to the DS save area. 


The debug store (DS) save area is a software-designated area of memory that is used to collect the following two 
types of information: 


¢ Branch records — When the BTS flag in the IA32_DEBUGCTL MSR is set, a branch record is stored in the BTS 
buffer in the DS save area whenever a taken branch, interrupt, or exception is detected. 


¢ PEBS records — When a performance counter is configured for PEBS, a PEBS record is stored in the PEBS 
buffer in the DS save area after the counter overflow occurs. This record contains the architectural state of the 
processor (state of the 8 general purpose registers, EIP register, and EFLAGS register) at the next occurrence 
of the PEBS event that caused the counter to overflow. When the state information has been logged, the 
counter is automatically reset to a specified value, and event counting begins again. The content layout of a 
PEBS record varies across different implementations that support PEBS. See Section 18.6.2.4.2 for details of 
enumerating PEBS record format. 


NOTES 


Prior to processors based on the Goldmont microarchitecture, PEBS facility only supports a subset 
of implementation-specific precise events. See Section 18.5.3.1 for a PEBS enhancement that can 
generate records for both precise and non-precise events. 


The DS save area and recording mechanism are disabled on INIT, processor Reset or transition to 
system-management mode (SMM) or IA-32e mode. It is similarly disabled on the generation of a 
machine-check exception on 45nm and 32nm Intel Atom processors and on processors with 
Netburst or Intel Core microarchitecture. 


The BTS and PEBS facilities may not be available on all processors. The availability of these facilities 
is indicated by the BTS_UNAVAILABLE and PEBS_UNAVAILABLE flags, respectively, in the 
IA32_MISC_ENABLE MSR (see Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 4). 


The DS save area is divided into three parts: buffer management area, branch trace store (BTS) buffer, and PEBS 
buffer (see Figure 17-5). The buffer management area is used to define the location and size of the BTS and PEBS 
buffers. The processor then uses the buffer management area to keep track of the branch and/or PEBS records in 
their respective buffers and to record the performance counter reset value. The linear address of the first byte of 
the DS buffer management area is specified with the IA32_DS_ AREA MSR. 


The fields in the buffer management area are as follows: 


¢ BTS buffer base — Linear address of the first byte of the BTS buffer. This address should point to a natural 
doubleword boundary. 


° BTS index — Linear address of the first byte of the next BTS record to be written to. Initially, this address 
should be the same as the address in the BTS buffer base field. 


¢ BTS absolute maximum — Linear address of the next byte past the end of the BTS buffer. This address should 
be a multiple of the BTS record size (12 bytes) plus 1. 
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BTS interrupt threshold — Linear address of the BTS record on which an interrupt is to be generated. This 
address must point to an offset from the BTS buffer base that is a multiple of the BTS record size. Also, it must 
be several records short of the BTS absolute maximum address to allow a pending interrupt to be handled prior 
to processor writing the BTS absolute maximum record. 

PEBS buffer base — Linear address of the first byte of the PEBS buffer. This address should point to a natural 
doubleword boundary. 

PEBS index — Linear address of the first byte of the next PEBS record to be written to. Initially, this address 
should be the same as the address in the PEBS buffer base field. 

PEBS absolute maximum — Linear address of the next byte past the end of the PEBS buffer. This address 
should be a multiple of the PEBS record size (40 bytes) plus 1. 

PEBS interrupt threshold — Linear address of the PEBS record on which an interrupt is to be generated. This 
address must point to an offset from the PEBS buffer base that is a multiple of the PEBS record size. Also, it 
must be several records short of the PEBS absolute maximum address to allow a pending interrupt to be 
handled prior to processor writing the PEBS absolute maximum record. 

PEBS counter reset value — A 64-bit value that the counter is to be set to when a PEBS record is written. Bits 
beyond the size of the counter are ignored. This value allows state information to be collected regularly every 
time the specified number of events occur. 


-— IA32_DS_AREA MSR 


DS Buffer Management Area BTS Buffer 


<< 
BTS Buffer Base OH—___> 


Branch Record 0 


BTS Index 4H 
BTS Absolute 8H 
Maximum 
BTS Interrupt Branch Record 1 
Threshold CH 
PEBS Buffer Base] 10H 
PEBS Index 14H 4 
PEBS Absolute 18H— 
| Maximum _| Branch Record n 
PEBS Interrupt |4cH 
Threshold _> 
PEBS on 
Counter Reset 24H PEBS Buffer 
R d 
eserve 30H PEBS Record 0 
PEBS Record 1 
|_y»| 
PEBS Record n 
oT 


Figure 17-5. DS Save Area Example! 


NOTES: 
1. This example represents the format for a system that supports PEBS on only one counter. 
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Figure 17-6 shows the structure of a 12-byte branch record in the BTS buffer. The fields in each record are as 

follows: 

¢ Last branch from — Linear address of the instruction from which the branch, interrupt, or exception was 
taken. 

¢ Last branch to — Linear address of the branch target or the first instruction in the interrupt or exception 
service routine. 

¢ Branch predicted — Bit 4 of field indicates whether the branch that was taken was predicted (set) or not 
predicted (clear). 


31 4 0 
Last Branch From OH 
Last Branch To 4H 
8H 
Branch Predicted __A 


Figure 17-6. 32-bit Branch Trace Record Format 


Figure 17-7 shows the structure of the 40-byte PEBS records. Nominally the register values are those at the begin- 
ning of the instruction that caused the event. However, there are cases where the registers may be logged ina 
partially modified state. The linear IP field shows the value in the EIP register translated from an offset into the 
current code segment to a linear address. 


31 0 
EFLAGS OH 
Linear IP 4H 

EAX 8H 
EBX CH 
ECX 10H 
EDX 14H 
ESI 18H 
EDI 1CH 
EBP 20H 
ESP 24H 


Figure 17-7. PEBS Record Format 


17.4.9.1 64 Bit Format of the DS Save Area 
When DTES64 = 1 (CPUID.1.ECX[2] = 1), the structure of the DS save area is shown in Figure 17-8. 


When DTES64 = 0 (CPUID.1.ECX[2] = 0) and IA-32e mode is active, the structure of the DS save area is shown in 
Figure 17-8. If IA-32e mode is not active the structure of the DS save area is as shown in Figure 17-5. 
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-— IA32_DS_AREA MSR 


DS Buffer Management Area BTS Buffer 


<< 
BTS Buffer Base OH 


Branch Record 0 


BTS Index 8H 
BTS Absolute 
Maximum 10H 
BTS Interrupt Branch Record 1 
Threshold 18H 
PEBS Buffer Base] 20H 
PEBS Index 28H —— 
“fom [oh 
|_PEBS Interruot | Branch Record n 
PEBS Interrupt | 38H 
Threshold L_» 
PEBS auH 
POE eee aee PEBS Buffer 
—S> 
Reserved 50H 


PEBS Record 0 


PEBS Record 1 


PEBS Record n 


L__yy 


Figure 17-8. IA-32e Mode DS Save Area Example! 


NOTES: 
1. This example represents the format for a system that supports PEBS on only one counter. 


The IA32_DS_AREA MSR holds the 64-bit linear address of the first byte of the DS buffer management area. The 
structure of a branch trace record is similar to that shown in Figure 17-6, but each field is 8 bytes in length. This 
makes each BTS record 24 bytes (see Figure 17-9). The structure of a PEBS record is similar to that shown in 
Figure 17-7, but each field is 8 bytes in length and architectural states include register R8 through R15. This makes 
the size of a PEBS record in 64-bit mode 144 bytes (see Figure 17-10). 


63 4 0 
Last Branch From 0H 
Last Branch To 8H 
10H 
Branch Predicted __A 


Figure 17-9. 64-bit Branch Trace Record Format 


Vol. 3B 17-21 


DEBUG, BRANCH PROFILE, TSC, AND INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) FEATURES 


63 0 

RFLAGS OH 
RIP 8H 
RAX 10H 
RBX 18H 
RCX 20H 
RDX 28H 
RSI 30H 
RDI 38H 
RBP 40H 
RSP 48H 
R8 50H 
Ris 88H 


Figure 17-10. 64-bit PEBS Record Format 


Fields in the buffer management area of a DS save area are described in Section 17.4.9. 


The format of a branch trace record and a PEBS record are the same as the 64-bit record formats shown in Figures 
17-9 and Figures 17-10, with the exception that the branch predicted bit is not supported by Intel Core microarchi- 
tecture or Intel Atom microarchitecture. The 64-bit record formats for BTS and PEBS apply to DS save area for all 
operating modes. 


The procedures used to program IA32_DEBUGCTL MSR to set up a BTS buffer or a CPL-qualified BTS are described 
in Section 17.4.9.3 and Section 17.4.9.4. 


Required elements for writing a DS interrupt service routine are largely the same on processors that support using 

DS Save area for BTS or PEBS records. However, on processors based on Intel NetBurst® microarchitecture, re- 

enabling counting requires writing to CCCRs. But a DS interrupt service routine on processors supporting architec- 

tural performance monitoring should: 

® Re-enable the enable bits in IA32_PERF_GLOBAL_CTRL MSR if it is servicing an overflow PMI due to PEBS. 

° Clear overflow indications by writing to IA32_PERF_GLOBAL_OVF_CTRL when a counting configuration is 
changed. This includes bit 62 (ClrOvfBuffer) and the overflow indication of counters used in either PEBS or 
general-purpose counting (specifically: bits 0 or 1; see Figures 18-3). 


17.4.9.2 Setting Up the DS Save Area 


To save branch records with the BTS buffer, the DS save area must first be set up in memory as described in the 
following procedure (See Section 18.6.2.4.1, “Setting up the PEBS Buffer,” for instructions for setting up a PEBS 
buffer, respectively, in the DS save area): 


1. Create the DS buffer management information area in memory (see Section 17.4.9, “BTS and DS Save Area,” 
and Section 17.4.9.1, “64 Bit Format of the DS Save Area”). Also see the additional notes in this section. 


Write the base linear address of the DS buffer management area into the IA32_DS_AREA MSR. 


Set up the performance counter entry in the xAPIC LVT for fixed delivery and edge sensitive. See Section 
10.5.1, “Local Vector Table.” 


4. Establish an interrupt handler in the IDT for the vector associated with the performance counter entry in the 
xAPIC LVT. 
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Write an interrupt service routine to handle the interrupt. See Section 17.4.9.5, “Writing the DS Interrupt 
Service Routine.” 


The following restrictions should be applied to the DS save area. 


The recording of branch records in the BTS buffer (or PEBS records in the PEBS buffer) may not operate 
properly if accesses to the linear addresses in any of the three DS save area sections cause page faults, VM 
exits, or the setting of accessed or dirty flags in the paging structures (ordinary or EPT). For that reason, 
system software should establish paging structures (both ordinary and EPT) to prevent such occurrences. 
Implications of this may be that an operating system should allocate this memory from a non-paged pool and 
that system software cannot do “lazy” page-table entry propagation for these pages. Some newer processor 
generations support “lazy” EPT page-table entry propagation for PEBS; see Section 18.3.10.1 and Section 
18.9.5 for more information. A virtual-machine monitor may choose to allow use of PEBS by guest software 
only if EPT maps all guest-physical memory as present and read/write. 


The DS save area can be larger than a page, but the pages must be mapped to contiguous linear addresses. 
The buffer may share a page, so it need not be aligned on a 4-KByte boundary. For performance reasons, the 
base of the buffer must be aligned on a doubleword boundary and should be aligned on a cache line boundary. 


It is recommended that the buffer size for the BTS buffer and the PEBS buffer be an integer multiple of the 
corresponding record sizes. 

The precise event records buffer should be large enough to hold the number of precise event records that can 
occur while waiting for the interrupt to be serviced. 


The DS save area should be in kernel space. It must not be on the same page as code, to avoid triggering self- 
modifying code actions. 


There are no memory type restrictions on the buffers, although it is recommended that the buffers be 
designated as WB memory type for performance considerations. 


Either the system must be prevented from entering A20M mode while DS save area is active, or bit 20 of all 
addresses within buffer bounds must be 0. 


Pages that contain buffers must be mapped to the same physical addresses for all processes, such that any 
change to control register CR3 will not change the DS addresses. 


The DS save area is expected to used only on systems with an enabled APIC. The LVT Performance Counter 
entry in the APCI must be initialized to use an interrupt gate instead of the trap gate. 


17.4.9.3. Setting Up the BTS Buffer 


Three flags in the MSR_DEBUGCTLA MSR (see Table 17-5), IA32_DEBUGCTL (see Figure 17-3), or 
MSR_DEBUGCTLB (see Figure 17-16) control the generation of branch records and storing of them in the BTS 
buffer; these are TR, BTS, and BTINT. The TR flag enables the generation of BTMs. The BTS flag determines 
whether the BTMs are sent out on the system bus (clear) or stored in the BTS buffer (set). BTMs cannot be simul- 
taneously sent to the system bus and logged in the BTS buffer. The BTINT flag enables the generation of an inter- 
rupt when the BTS buffer is full. When this flag is clear, the BTS buffer is a circular buffer. 


Table 17-5. IA32_DEBUGCTL Flag Encodings 


TR BTS BTINT Description 

0 X X Branch trace messages (BTMs) off 

1 0 X Generate BTMs 

1 1 0 Store BTMs in the BTS buffer, used here as a circular buffer 

1 1 1 Store BTMs in the BTS buffer, and generate an interrupt when the buffer is nearly full 


The following procedure describes how to set up a DS Save area to collect branch records in the BTS buffer: 


1. 


Place values in the BTS buffer base, BTS index, BTS absolute maximum, and BTS interrupt threshold fields of 
the DS buffer management area to set up the BTS buffer in memory. 


Set the TR and BTS flags in the IA32_DEBUGCTL for Intel Core Solo and Intel Core Duo processors or later 
processors (or MSR_DEBUGCTLA MSR for processors based on Intel NetBurst Microarchitecture; or 
MSR_DEBUGCTLB for Pentium M processors). 
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3. Clear the BTINT flag in the corresponding IA32_DEBUGCTL (or MSR_DEBUGCTLA MSR; or MSR_DEBUGCTLB) 
if a circular BTS buffer is desired. 


NOTES 


If the buffer size is set to less than the minimum allowable value (i.e. BTS absolute maximum < 1 
+ size of BTS record), the results of BTS is undefined. 


In order to prevent generating an interrupt, when working with circular BTS buffer, SW need to set 
BTS interrupt threshold to a value greater than BTS absolute maximum (fields of the DS buffer 
management area). It's not enough to clear the BTINT flag itself only. 


17.4.9.4 Setting Up CPL-Qualified BTS 


If the processor supports CPL-qualified last branch recording mechanism, the generation of branch records and 
storing of them in the BTS buffer are determined by: TR, BTS, BTS_OFF_OS, BTS_OFF_USR, and BTINT. The 
encoding of these five bits are shown in Table 17-6. 
Table 17-6. CPL-Qualified Branch Trace Store Encodings 
TR BTS BTS_OFF_OS |BTS_OFF_USR | BTINT Description 
Branch trace messages (BTMs) off 
Generates BTMs but do not store BTMs 
Store all BTMs in the BTS buffer, used here as a circular buffer 
Store BTMs with CPL > 0 in the BTS buffer 
Store BTMs with CPL = 0 in the BTS buffer 
Generate BTMs but do not store BTMs 


Store all BTMs in the BTS buffer; generate an interrupt when the 
buffer is nearly full 


1 1 1 0 1 Store BTMs with CPL > 0 in the BTS buffer; generate an interrupt 
when the buffer is nearly full 


1 1 0) 1 1 Store BTMs with CPL = 0 in the BTS buffer; generate an interrupt 
when the buffer is nearly full 


oO;| —| OO} —| CO} X<| 
o;| —| —| O}] CO} X<| 
—| <;}| CO}; OO] CO} X<| << 


17.4.9.5 Writing the DS Interrupt Service Routine 


The BTS, non-precise event-based sampling, and PEBS facilities share the same interrupt vector and interrupt 
service routine (called the debug store interrupt service routine or DS ISR). To handle BTS, non-precise event- 
based sampling, and PEBS interrupts: separate handler routines must be included in the DS ISR. Use the following 
guidelines when writing a DS ISR to handle BTS, non-precise event-based sampling, and/or PEBS interrupts. 


¢ The DS interrupt service routine (ISR) must be part of a kernel driver and operate at a current privilege level of 
0 to secure the buffer storage area. 


¢ Because the BTS, non-precise event-based sampling, and PEBS facilities share the same interrupt vector, the 
DS ISR must check for all the possible causes of interrupts from these facilities and pass control on to the 
appropriate handler. 


BTS and PEBS buffer overflow would be the sources of the interrupt if the buffer index matches/exceeds the 
interrupt threshold specified. Detection of non-precise event-based sampling as the source of the interrupt is 
accomplished by checking for counter overflow. 


¢ There must be separate save areas, buffers, and state for each processor in an MP system. 


¢ Upon entering the ISR, branch trace messages and PEBS should be disabled to prevent race conditions during 
access to the DS save area. This is done by clearing TR flag in the IA32_ DEBUGCTL (or MSR_DEBUGCTLA MSR) 
and by clearing the precise event enable flag in the MSR_PEBS_ENABLE MSR. These settings should be 
restored to their original values when exiting the ISR. 
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¢ The processor will not disable the DS save area when the buffer is full and the circular mode has not been 
selected. The current DS setting must be retained and restored by the ISR on exit. 


e ~=6After reading the data in the appropriate buffer, up to but not including the current index into the buffer, the ISR 
must reset the buffer index to the beginning of the buffer. Otherwise, everything up to the index will look like 
new entries upon the next invocation of the ISR. 


¢ The ISR must clear the mask bit in the performance counter LVT entry. 


¢ The ISR must re-enable the counters to count via IA32_PERF_GLOBAL_CTRL/IA32_PERF_GLOBAL_OVF_CTRL 
if it is servicing an overflow PMI due to PEBS (or via CCCR's ENABLE bit on processor based on Intel NetBurst 
microarchitecture). 


° The Pentium 4 Processor and Intel Xeon Processor mask PMIs upon receiving an interrupt. Clear this condition 
before leaving the interrupt handler. 


17.5 LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING (INTEL’ CORE™ 2 
DUO AND INTEL*° ATOM™ PROCESSORS) 


The Intel Core 2 Duo processor family and Intel Xeon processors based on Intel Core microarchitecture or 
enhanced Intel Core microarchitecture provide last branch interrupt and exception recording. The facilities 
described in this section also apply to 45 nm and 32 nm Intel Atom processors. These capabilities are similar to 
those found in Pentium 4 processors, including support for the following facilities: 


¢ Debug Trace and Branch Recording Control — The IA32_DEBUGCTL MSR provide bit fields for software to 
configure mechanisms related to debug trace, branch recording, branch trace store, and performance counter 
operations. See Section 17.4.1 for a description of the flags. See Figure 17-3 for the MSR layout. 


¢ Last branch record (LBR) stack — There are a collection of MSR pairs that store the source and destination 
addresses related to recently executed branches. See Section 17.5.1. 


* Monitoring and single-stepping of branches, exceptions, and interrupts 


— See Section 17.4.2 and Section 17.4.3. In addition, the ability to freeze the LBR stack on a PMI request is 
available. 


— 45 nm and 32 nm Intel Atom processors clear the TR flag when the FREEZE_LBRS_ON_PMI flag is set. 
° Branch trace messages — See Section 17.4.4. 
° Last exception records — See Section 17.13.3. 
¢ Branch trace store and CPL-qualified BTS — See Section 17.4.5. 
¢ FREEZE_LBRS_ON_PMI flag (bit 11) — see Section 17.4.7 for legacy Freeze_LBRs_On_PMI operation. 


¢ FREEZE_PERFMON_ON_PMI flag (bit 12) — see Section 17.4.7 for legacy Freeze_Perfmon_On_PMI 
operation. 


¢ FREEZE_WHILE_SMM (bit 14) — FREEZE_WHILE_SMM is supported if 
IA32_PERF_CAPABILITIES.FREEZE_WHILE_SMM[Bit 12] is reporting 1. See Section 17.4.1. 


17.5.1 LBR Stack 


The last branch record stack and top-of-stack (TOS) pointer MSRs are supported across Intel Core 2, Intel Atom 
processor families, and Intel processors based on Intel NetBurst microarchitecture. 


Four pairs of MSRs are supported in the LBR stack for Intel Core 2 processors families and Intel processors based 
on Intel NetBurst microarchitecture: 


¢ Last Branch Record (LBR) Stack 


— MSR_LASTBRANCH_0O_FROM_IP (address 40H) through MSR_LASTBRANCH_3_FROM_IP (address 43H) 
store source addresses 


— MSR_LASTBRANCH_0_TO_IP (address 60H) through MSR_LASTBRANCH_3_TO_IP (address 63H) store 
destination addresses 
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¢ Last Branch Record Top-of-Stack (TOS) Pointer — The lowest significant 2 bits of the TOS Pointer MSR 
(MSR_LASTBRANCH_TOS, address 1C9H) contains a pointer to the MSR in the LBR stack that contains the most 
recent branch, interrupt, or exception recorded. 


Eight pairs of MSRs are supported in the LBR stack for 45 nm and 32 nm Intel Atom processors: 
¢ Last Branch Record (LBR) Stack 


— MSR_LASTBRANCH_0O_FROM_IP (address 40H) through MSR_LASTBRANCH_7_FROML_IP (address 47H) 
store source addresses 


— MSR_LASTBRANCH_0O_TO_IP (address 60H) through MSR_LASTBRANCH_7_TO_IP (address 67H) store 
destination addresses 


¢ Last Branch Record Top-of-Stack (TOS) Pointer — The lowest significant 3 bits of the TOS Pointer MSR 
(MSR_LASTBRANCH_TOS, address 1C9H) contains a pointer to the MSR in the LBR stack that contains the most 
recent branch, interrupt, or exception recorded. 


The address format written in the FROM_IP/TO_IP MSRS may differ between processors. Software should query 
IA32_PERF_CAPABILITIES[5:0] and consult Section 17.4.8.1. The behavior of the MSR_LER_TO_LIP and the 
MSR_LER_FROM_LIP MSRs corresponds to that of the LastExceptionToIP and LastExceptionFromIP MSRs found in 
P6 family processors. 


17.5.2 LBR Stack in Intel Atom Processors based on the Silvermont Microarchitecture 


The last branch record stack and top-of-stack (TOS) pointer MSRs are supported in Intel Atom processors based on 
the Silvermont and Airmont microarchitectures. Eight pairs of MSRs are supported in the LBR stack. 


LBR filtering is supported. Filtering of LBRs based on a combination of CPL and branch type conditions is supported. 
When LBR filtering is enabled, the LBR stack only captures the subset of branches that are specified by 
MSR_LBR_SELECT. The layout of MSR_LBR_SELECT is described in Table 17-11. 


17.6 LAST BRANCH, CALL STACK, INTERRUPT, AND EXCEPTION RECORDING 
FOR PROCESSORS BASED ON GOLDMONT MICROARCHITECTURE 


Processors based on the Goldmont microarchitecture extend the capabilities described in Section 17.5.2 with the 
following enhancements: 


¢* Supports new LBR format encoding 00110b in IA32_PERF_CAPABILITIES[5:0]. 


° Size of LBR stack increased to 32. Each entry includes MSR_LASTBRANCH_x_FROM_IP (address 0x680..0x69f) 
and MSR_LASTBRANCH_x_TO_IP (address 0x6c0..0x6df). 


e LBR call stack filtering supported. The layout of MSR_LBR_SELECT is described in Table 17-13. 
e Elapsed cycle information is added to MSR_LASTBRANCH_x_TO_IP. Format is shown in Table 17-7. 


® Misprediction info is reported in the upper bits of MSR_LASTBRANCH_x_FROM_IP. MISPRED bit format is 
shown in Table 17-8. 


e Streamlined Freeze_LBRs_On_PMI operation; see Section 17.12.2. 


e LBR MSRs may be cleared when MWAIT is used to request a C-state that is numerically higher than Ci; see 
Section 17.12.3. 


Table 17-7. MSR_LASTBRANCH_x_TO_IP for the Goldmont Microarchitecture 


Bit Field Bit Offset | Access |Description 

Data 47:0 R/W This is the “branch to” address. See Section 17.4.8.1 for address format. 
Cycle Count 63:48 R/W Elapsed core clocks since last update to the LBR stack. 

(Saturating) 
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17.7. LAST BRANCH, CALL STACK, INTERRUPT, AND EXCEPTION RECORDING 
FOR PROCESSORS BASED ON GOLDMONT PLUS MICROARCHITECTURE 


Next generation Intel Atom processors are based on the Goldmont Plus microarchitecture. Processors based on the 
Goldmont Plus microarchitecture extend the capabilities described in Section 17.6 with the following changes: 
e Enumeration of new LBR format: encoding 00111b in IA32_PERF_CAPABILITIES[5:0] is supported, see 
Section 17.4.8.1. 
e Each LBR stack entry consists of three MSRs: 

— MSR_LASTBRANCH_x_FROM_IP, the layout is simplified, see Table 17-9. 

— MSR_LASTBRANCH_x_TO_IP, the layout is the same as Table 17-9. 


— MSR_LBR_INFO_x, stores branch prediction flag, TSX info, and elapsed cycle data. Layout is the same as 
Table 17-16. 


17.8. LAST BRANCH, INTERRUPT AND EXCEPTION RECORDING FOR INTEL® 
XEON PHI” PROCESSOR 7200/5200/3200 


The last branch record stack and top-of-stack (TOS) pointer MSRs are supported in the Intel® Xeon Phi™ processor 
7200/5200/3200 series based on the Knights Landing microarchitecture. Eight pairs of MSRs are supported in the 
LBR stack, per thread: 

¢ Last Branch Record (LBR) Stack 


— MSR_LASTBRANCH_0_FROM_IP (address 680H) through MSR_LASTBRANCH_7_FROM_IP (address 687H) 
store source addresses. 


— MSR_LASTBRANCH_0_TO_IP (address 6COH) through MSR_LASTBRANCH_7_TO_IP (address 6C7H) store 
destination addresses. 
¢ Last Branch Record Top-of-Stack (TOS) Pointer — The lowest significant 3 bits of the TOS Pointer MSR 
(MSR_LASTBRANCH_TOS, address 1C9H) contains a pointer to the MSR in the LBR stack that contains the 
most recent branch, interrupt, or exception recorded. 


LBR filtering is supported. Filtering of LBRs based on a combination of CPL and branch type conditions is supported. 
When LBR filtering is enabled, the LBR stack only captures the subset of branches that are specified by 
MSR_LBR_SELECT. The layout of MSR_LBR_SELECT is described in Table 17-11. 


The address format written in the FROM_IP/TO_IP MSRS may differ between processors. Software should query 
IA32_PERF_CAPABILITIES[5:0] and consult Section 17.4.8.1.The behavior of the MSR_LER_TO_LIP and the 
MSR_LER_FROM_LIP MSRs corresponds to that of the LastExceptionToIP and LastExceptionFromIP MSRs found in 
the P6 family processors. 


17.9 LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING FOR 
PROCESSORS BASED ON INTEL® MICROARCHITECTURE CODE NAME 
NEHALEM 


The processors based on Intel® microarchitecture code name Nehalem and Intel® microarchitecture code name 
Westmere support last branch interrupt and exception recording. These capabilities are similar to those found in 
Intel Core 2 processors and adds additional capabilities: 


¢ Debug Trace and Branch Recording Control — The IA32_DEBUGCTL MSR provides bit fields for software to 
configure mechanisms related to debug trace, branch recording, branch trace store, and performance counter 
operations. See Section 17.4.1 for a description of the flags. See Figure 17-11 for the MSR layout. 


¢ Last branch record (LBR) stack — There are 16 MSR pairs that store the source and destination addresses 
related to recently executed branches. See Section 17.9.1. 
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° Monitoring and single-stepping of branches, exceptions, and interrupts — See Section 17.4.2 and 
Section 17.4.3. In addition, the ability to freeze the LBR stack on a PMI request is available. 


¢ Branch trace messages — The IA32_DEBUGCTL MSR provides bit fields for software to enable each logical 
processor to generate branch trace messages. See Section 17.4.4. However, not all BTM messages are 
observable using the Intel® QPI link. 


° Last exception records — See Section 17.13.3. 
¢ Branch trace store and CPL-qualified BTS — See Section 17.4.6 and Section 17.4.5. 
¢ FREEZE_LBRS_ON_PMI flag (bit 11) — see Section 17.4.7 for legacy Freeze_LBRs_On_PMI operation. 


¢ FREEZE_PERFMON_ON_PMI flag (bit 12) — see Section 17.4.7 for legacy Freeze_Perfmon_On_PMI 
operation. 


¢ UNCORE_PMI_EN (bit 13) — When set. this logical processor is enabled to receive an counter overflow 
interrupt form the uncore. 


° FREEZE_WHILE_SMM (bit 14) — FREEZE_WHILE_SMM is supported if 
IA32_PERF_CAPABILITIES.FREEZE_WHILE_SMM[Bit 12] is reporting 1. See Section 17.4.1. 
Processors based on Intel microarchitecture code name Nehalem provide additional capabilities: 


° Independent control of uncore PMI — The IA32_DEBUGCTL MSR provides a bit field (see Figure 17-11) for 
software to enable each logical processor to receive an uncore counter overflow interrupt. 


¢ LBR filtering — Processors based on Intel microarchitecture code name Nehalem support filtering of LBR 
based on combination of CPL and branch type conditions. When LBR filtering is enabled, the LBR stack only 
captures the subset of branches that are specified by MSR_LBR_SELECT. 


31 141312111098 7654321 0 


Reserved 


FREEZE_WHILE_SMM 
UNCORE_PMI_EN 

FREEZE_PERFMON_ON_PMI 
FREEZE_LBRS_ON_PMI 
BTS_OFF_USR — BTS off in user code 
BTS_OFF_OS — BTS off in OS 
BTINT — Branch trace interrupt 
BTS — Branch trace store 

TR — Trace messages enable 
Reserved 
BTF — Single-step on branches 
LBR — Last branch/interrupt/exception 


Figure 17-11. [A32_DEBUGCTL MSR for Processors based 
on Intel microarchitecture code name Nehalem 
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17.9.1 LBR Stack 


Processors based on Intel microarchitecture code name Nehalem provide 16 pairs of MSR to record last branch 
record information. The layout of each MSR pair is shown in Table 17-8 and Table 17-9. 


Table 17-8. MSR_LASTBRANCH_x_FROM_IP 


Bit Field Bit Offset |Access |Description 

Data 47:0 R/W This is the “branch from” address. See Section 17.4.8.1 for address format. 

SIGN_EXt 62:48 R/W Signed extension of bit 47 of this register. 

MISPRED 63 R/W When set, indicates either the target of the branch was mispredicted and/or the 
direction (taken/non-taken) was mispredicted; otherwise, the target branch was 
predicted. 

Table 17-9. MSR_LASTBRANCH_x_TO_IP 

Bit Field Bit Offset | Access |Description 

Data 47:0 R/W This is the “branch to” address. See Section 17.4.8.1 for address format 

SIGN_EXt 63:48 R/W Signed extension of bit 47 of this register. 


Processors based on Intel microarchitecture code name Nehalem have an LBR MSR Stack as shown in Table 17-10. 


Table 17-10. LBR Stack Size and TOS Pointer Range 


DisplayFamily_DisplayModel 


Size of LBR Stack 


Range of TOS Pointer 


06_1AH 


16 


Oto15 


17.9.2 Filtering of Last Branch Records 


MSR_LBR_SELECT is cleared to zero at RESET, and LBR filtering is disabled, i.e. all branches will be captured. 
MSR_LBR_SELECT provides bit fields to specify the conditions of subsets of branches that will not be captured in 


the LBR. The layout of MSR_LBR_SELECT is shown in Table 17-11. 


Table 17-11. MSR_LBR_SELECT for Intel microarchitecture code name Nehalem 


Bit Field Bit Offset |Access |Description 

CPL_EQ_O 0 R/W When set, do not capture branches ending in ring O 
CPL_NEQ_O 1 R/W When set, do not capture branches ending in ring >O 
jcc 2 R/W When set, do not capture conditional branches 
NEAR_REL_CALL | 3 R/W When set, do not capture near relative calls 
NEAR_IND_CALL /|4 R/W When set, do not capture near indirect calls 
NEAR_RET 5 R/W When set, do not capture near returns 
NEAR_IND_JMP 6 R/W When set, do not capture near indirect jumps 
NEAR_REL_JMP 7 R/W When set, do not capture near relative jumps 
FAR_BRANCH 8 R/W When set, do not capture far branches 


Reserved 63:9 


Must be zero 
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17.10 LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING FOR 
PROCESSORS BASED ON INTEL® MICROARCHITECTURE CODE NAME 
SANDY BRIDGE 


Generally, all of the last branch record, interrupt and exception recording facility described in Section 17.9, “Last 
Branch, Interrupt, and Exception Recording for Processors based on Intel® Microarchitecture code name 
Nehalem”, apply to processors based on Intel microarchitecture code name Sandy Bridge. For processors based on 
Intel microarchitecture code name Ivy Bridge, the same holds true. 


One difference of note is that MSR_LBR_SELECT is shared between two logical processors in the same core. In Intel 
microarchitecture code name Sandy Bridge, each logical processor has its own MSR_LBR_SELECT. The filtering 
semantics for “Near_ind_jmp” and “Near_rel_jmp” has been enhanced, see Table 17-12. 


Table 17-12. MSR_LBR_SELECT for Intel® microarchitecture code name Sandy Bridge 


Bit Field Bit Offset | Access |Description 

CPL_EQ_O 0 R/W When set, do not capture branches ending in ring O 

CPL_NEQ_O 1 R/W When set, do not capture branches ending in ring >0 

Jcc 2 R/W When set, do not capture conditional branches 

NEAR_REL_CALL |3 R/W When set, do not capture near relative calls 

NEAR_IND_CALL |4 R/W When set, do not capture near indirect calls 

NEAR_RET 5 R/W When set, do not capture near returns 

NEAR_IND_JMP 6 R/W When set, do not capture near indirect jumps except near indirect calls and near returns 
NEAR_REL_JMP 7 R/W When set, do not capture near relative jumps except near relative calls. 
FAR_BRANCH 8 R/W When set, do not capture far branches 

Reserved 63:9 Must be zero 


17.11 LAST BRANCH, CALL STACK, INTERRUPT, AND EXCEPTION RECORDING 
FOR PROCESSORS BASED ON HASWELL MICROARCHITECTURE 


Generally, all of the last branch record, interrupt and exception recording facility described in Section 17.10, “Last 
Branch, Interrupt, and Exception Recording for Processors based on Intel® Microarchitecture code name Sandy 
Bridge”, apply to next generation processors based on Intel microarchitecture code name Haswell. 


The LBR facility also supports an alternate capability to profile call stack profiles. Configuring the LBR facility to 
conduct call stack profiling is by writing 1 to the MSR_LBR_SELECT.EN_CALLSTACK[bit 9]; see Table 17-13. If 
MSR_LBR_SELECT.EN_CALLSTACK is clear, the LBR facility will capture branches normally as described in Section 
17.10. 


Table 17-13. MSR_LBR_SELECT for Intel® microarchitecture code name Haswell 


Bit Field Bit Offset | Access |Description 

CPL_EQ_O 0 R/W When set, do not capture branches ending in ring O 

CPL_NEQ_O 1 R/W When set, do not capture branches ending in ring >O 

Jcc 2 R/W When set, do not capture conditional branches 

NEAR_REL_CALL |3 R/W When set, do not capture near relative calls 

NEAR_IND_CALL |4 R/W When set, do not capture near indirect calls 

NEAR_RET 5 R/W When set, do not capture near returns 

NEAR_IND_JMP 6 R/W When set, do not capture near indirect jumps except near indirect calls and near returns 
NEAR_REL_JMP 7 R/W When set, do not capture near relative jumps except near relative calls. 
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Table 17-13. MSR_LBR_SELECT for Intel® microarchitecture code name Haswell 


Bit Field Bit Offset | Access |Description 

FAR_BRANCH 8 R/W When set, do not capture far branches 

EN_CALLSTACK' |9 Enable LBR stack to use LIFO filtering to capture Call stack profile 
Reserved 63:10 Must be zero 

NOTES: 


1. Must set valid combination of bits 0-8 in conjunction with bit 9 (as described below), otherwise the contents of the LBR MSRs are 
undefined. 


The call stack profiling capability is an enhancement of the LBR facility. The LBR stack is a ring buffer typically used 
to profile control flow transitions resulting from branches. However, the finite depth of the LBR stack often become 
less effective when profiling certain high-level languages (e.g. C++), where a transition of the execution flow is 
accompanied by a large number of leaf function calls, each of which returns an individual parameter to form the list 
of parameters for the main execution function call. A long list of such parameters returned by the leaf functions 
would serve to flush the data captured in the LBR stack, often losing the main execution context. 


When the call stack feature is enabled, the LBR stack will capture unfiltered call data normally, but as return 
instructions are executed the last captured branch record is flushed from the on-chip registers in a last-in first-out 
(LIFO) manner. Thus, branch information relative to leaf functions will not be captured, while preserving the call 
stack information of the main line execution path. 


The configuration of the call stack facility is summarized below: 


° Set IA32_DEBUGCTL.LBR (bit 0) to enable the LBR stack to capture branch records. The source and target 
addresses of the call branches will be captured in the 16 pairs of From/To LBR MSRs that form the LBR stack. 


¢ Program the Top of Stack (TOS) MSR that points to the last valid from/to pair. This register is incremented by 
1, modulo 16, before recording the next pair of addresses. 


¢ Program the branch filtering bits of MSR_LBR_SELECT (bits 0:8) as desired. 
¢ Program the MSR_LBR_SELECT to enable LIFO filtering of return instructions with: 


— The following bits in MSR_LBR_SELECT must be set to ‘1’: JCC, NEAR_IND_JMP, NEAR_REL_JMP, 
FAR_BRANCH, EN_CALLSTACK; 


— The following bits in MSR_LBR_SELECT must be cleared: NEAR_REL_CALL, NEAR-IND_CALL, NEAR_RET; 
— At most one of CPL_EQ_0, CPL_NEQ_0 is set. 


Note that when call stack profiling is enabled, “zero length calls” are excluded from writing into the LBRs. (A “zero 
length call” uses the attribute of the call instruction to push the immediate instruction pointer on to the stack and 
then pops off that address into a register. This is accomplished without any matching return on the call.) 


17.11.1. ULBR Stack Enhancement 


Processors based on Intel microarchitecture code name Haswell provide 16 pairs of MSR to record last branch 
record information. The layout of each MSR pair is enumerated by IA32_PERF_CAPABILITIES[5:0] = 04H, and is 
shown in Table 17-14 and Table 17-9. 


Table 17-14. MSR_LASTBRANCH_x_FROM_IP with TSX Information 


Bit Field Bit Offset |Access |Description 

Data 47:0 R/W This is the “branch from” address. See Section 17.4.8.1 for address format. 
SIGN_EXT 60:48 R/W Signed extension of bit 47 of this register. 

TSX_ABORT 61 R/W When set, indicates a TSX Abort entry 


LBR_FROM: EIP at the time of the TSX Abort 
LBR_TO: EIP of the start of HLE region, or EIP of the RTM Abort Handler 


IN_TSX 62 R/W When set, indicates the entry occurred in a TSX region 
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Table 17-14. MSR_LASTBRANCH_x_FROM_IP with TSX Information (Contd.) 


Bit Field Bit Offset | Access |Description 

MISPRED 63 R/W When set, indicates either the target of the branch was mispredicted and/or the 
direction (taken/non-taken) was mispredicted; otherwise, the target branch was 
predicted. 


17.12 LAST BRANCH, CALL STACK, INTERRUPT, AND EXCEPTION RECORDING 
FOR PROCESSORS BASED ON SKYLAKE MICROARCHITECTURE 


Processors based on the Skylake microarchitecture provide a number of enhancement with storing last branch 
records: 


® enumeration of new LBR format: encoding 00101b in IA32_PERF_CAPABILITIES[5:0] is supported, see Section 
17.4.8.1. 


¢ Each LBR stack entry consists of a triplets of MSRs: 

— MSR_LASTBRANCH_x_FROM_IP, the layout is simplified, see Table 17-9. 

— MSR_LASTBRANCH_x_TO_IP, the layout is the same as Table 17-9. 

— MSR_LBR_INFO_x, stores branch prediction flag, TSX info, and elapsed cycle data. 
° Size of LBR stack increased to 32. 


Processors based on the Skylake microarchitecture supports the same LBR filtering capabilities as described in 
Table 17-13. 


Table 17-15. LBR Stack Size and TOS Pointer Range 


DisplayFamily_DisplayModel Size of LBR Stack Range of TOS Pointer 


06_4€H, 06_5€H 32 0 to 31 


17.12.1 MSR_LBR_INFO_x MSR 
The layout of each MSR_LBR_INFO_x MSR is shown in Table 17-16. 


Table 17-16. MSR_LBR_INFO_x 


Bit Field Bit Offset | Access |Description 

Cycle Count 15:0 R/W Elapsed core clocks since last update to the LBR stack. 
(saturating) 

Reserved 60:16 R/W Reserved 

TSX_ABORT 61 R/W When set, indicates a TSX Abort entry 


LBR_FROM: EIP at the time of the TSX Abort 
LBR_TO: EIP of the start of HLE region OR 
EIP of the RTM Abort Handler 


IN_TSX 62 R/W When set, indicates the entry occurred in a TSX region. 

MISPRED 63 R/W When set, indicates either the target of the branch was mispredicted and/or the 
direction (taken/non-taken) was mispredicted; otherwise, the target branch was 
predicted. 
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17.12.2 Streamlined Freeze_LBRs_On_PMI Operation 


The FREEZE_LBRS_ON_PMI feature causes the LBRs to be frozen on a hardware request for a PMI. This prevents 
the LBRs from being overwritten by new branches, allowing the PMI handler to examine the control flow that 
preceded the PMI generation. Architectural performance monitoring version 4 and above supports a streamlined 
FREEZE_LBRs_ON_PMI operation for PMI service routine that replaces the legacy FREEZE_LBRs_ON_PMI opera- 
tion (see Section 17.4.7). 


While the legacy FREEZE_LBRS_ON_PMI clear the LBR bit in the IA32_DEBUGCTL MSR on a PMI request, the 
streamlined FREEZE_LBRS_ON_PMI will set the LBR_FRZ bit in IA32_PERF_GLOBAL_STATUS. Branches will not 
cause the LBRs to be updated when LBR_FRZ is set. Software can clear LBR_FRZ at the same time as it clears over- 
flow bits by setting the LBR_FRZ bit as well as the needed overflow bit when writing to 
IA32_PERF_GLOBAL_STATUS_RESET MSR. 


This streamlined behavior avoids race conditions between software and processor writes to IA32_DEBUGCTL that 
are possible with FREEZE_LBRS_ON_PMI clearing of the LBR enable. 


17.12.3 LBR Behavior and Deep C-State 


When MWAIT is used to request a C-state that is numerically higher than C1, then LBR state may be initialized to 
zero depending on optimized “waiting” state that is selected by the processor The affected LBR states include the 
FROM, TO, INFO, LAST_BRANCH, LER and LBR_TOS registers. The LBR enable bit and LBR_FROZEN bit are not 
affected. The LBR-time of the first LBR record inserted after an exit from such a C-state request will be zero. 


17.13. LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING (PROCESSORS 
BASED ON INTEL NETBURST® MICROARCHITECTURE) 


Pentium 4 and Intel Xeon processors based on Intel NetBurst microarchitecture provide the following methods for 

recording taken branches, interrupts and exceptions: 

¢ Store branch records in the last branch record (LBR) stack MSRs for the most recent taken branches, 
interrupts, and/or exceptions in MSRs. A branch record consist of a branch-from and a branch-to instruction 
address. 

¢ Send the branch records out on the system bus as branch trace messages (BTMs). 

¢ Log BTMs in a memory-resident branch trace store (BTS) buffer. 

To support these functions, the processor provides the following MSRs and related facilities: 

¢ MSR_DEBUGCTLA MSR — Enables last branch, interrupt, and exception recording; single-stepping on taken 
branches; branch trace messages (BTMs); and branch trace store (BTS). This register is named DebugCtIMSR 
in the P6 family processors. 

° Debug store (DS) feature flag (CPUID.1:EDX.DS[bit 21]) — Indicates that the processor provides the 
debug store (DS) mechanism, which allows BTMs to be stored in a memory-resident BTS buffer. 

° CPL-qualified debug store (DS) feature flag (CPUID.1:ECX.DS-CPL[bit 4]) — Indicates that the 
processor provides a CPL-qualified debug store (DS) mechanism, which allows software to selectively skip 
sending and storing BTMs, according to specified current privilege level settings, into a memory-resident BTS 
buffer. 

° IA32_MISC_ENABLE MSR — Indicates that the processor provides the BTS facilities. 

¢ Last branch record (LBR) stack — The LBR stack is a circular stack that consists of four MSRs 
(MSR_LASTBRANCH_0O through MSR_LASTBRANCH_3) for the Pentium 4 and Intel Xeon processor family 
[CPUID family OFH, models OH-02H]. The LBR stack consists of 16 MSR pairs 
(MSR_LASTBRANCH_0O_FROM_IP through MSR_LASTBRANCH_15_FROM_IP and 
MSR_LASTBRANCH_0_TO_IP through MSR_LASTBRANCH_15_TO_IP) for the Pentium 4 and Intel Xeon 
processor family [CPUID family OFH, model 03H]. 

¢ Last branch record top-of-stack (TOS) pointer — The TOS Pointer MSR contains a 2-bit pointer (0-3) to 
the MSR in the LBR stack that contains the most recent branch, interrupt, or exception recorded for the 
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Pentium 4 and Intel Xeon processor family [CPUID family OFH, models OH-02H]. This pointer becomes a 4-bit 
pointer (0-15) for the Pentium 4 and Intel Xeon processor family [CPUID family OFH, model 03H]. See also: 
Table 17-17, Figure 17-12, and Section 17.13.2, “LBR Stack for Processors Based on Intel NetBurst® Microar- 
chitecture.” 


Last exception record — See Section 17.13.3, “Last Exception Records.” 


17.13.1 MSR_DEBUGCTLA MSR 


The MSR_DEBUGCTLA MSR enables and disables the various last branch recording mechanisms described in the 
previous section. This register can be written to using the WRMSR instruction, when operating at privilege level 0 
or when in real-address mode. A protected-mode operating system procedure is required to provide user access to 
this register. Figure 17-12 shows the flags in the MSR_DEBUGCTLA MSR. The functions of these flags are as 
follows: 


LBR (last branch/interrupt/exception) flag (bit 0) — When set, the processor records a running trace of 
the most recent branches, interrupts, and/or exceptions taken by the processor (prior to a debug exception 
being generated) in the last branch record (LBR) stack. Each branch, interrupt, or exception is recorded as a 
64-bit branch record. The processor clears this flag whenever a debug exception is generated (for example, 
when an instruction or data breakpoint or a single-step trap occurs). See Section 17.13.2, “LBR Stack for 
Processors Based on Intel NetBurst® Microarchitecture.” 


BTF (single-step on branches) flag (bit 1) — When set, the processor treats the TF flag in the EFLAGS 
register as a “single-step on branches” flag rather than a “single-step on instructions” flag. This mechanism 
allows single-stepping the processor on taken branches. See Section 17.4.3, “Single-Stepping on Branches.” 


TR (trace message enable) flag (bit 2) — When set, branch trace messages are enabled. Thereafter, when 
the processor detects a taken branch, interrupt, or exception, it sends the branch record out on the system bus 
as a branch trace message (BTM). See Section 17.4.4, “Branch Trace Messages.” 


31 76543210 


Reserved 


BTS_OFF_USR — Disable storing non-CPL_0 BTS — 
BTS_OFF_OS — Disable storing CPL_O BTS 

BTINT — Branch trace interrupt 
BTS — Branch trace store 
TR — Trace messages enable 
BTF — Single-step on branches 
LBR — Last branch/interrupt/exception 


Figure 17-12. MSR_DEBUGCTLA MSR for Pentium 4 and Intel Xeon Processors 


BTS (branch trace store) flag (bit 3) — When set, enables the BTS facilities to log BTMs to a memory- 
resident BTS buffer that is part of the DS save area. See Section 17.4.9, “BTS and DS Save Area.” 


BTINT (branch trace interrupt) flag (bits 4) — When set, the BTS facilities generate an interrupt when the 
BTS buffer is full. When clear, BTMs are logged to the BTS buffer in a circular fashion. See Section 17.4.5, “Branch 
Trace Store (BTS).” 

BTS_OFF_OS (disable ring 0 branch trace store) flag (bit 5) — When set, enables the BTS facilities to 
skip sending/logging CPL_O BTMs to the memory-resident BTS buffer. See Section 17.13.2, “LBR Stack for 
Processors Based on Intel NetBurst® Microarchitecture.” 

BTS_OFF_USR (disable ring O branch trace store) flag (bit 6) — When set, enables the BTS facilities to 
skip sending/logging non-CPL_O BTMs to the memory-resident BTS buffer. See Section 17.13.2, “LBR Stack for 
Processors Based on Intel NetBurst® Microarchitecture.” 
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NOTE 


The initial implementation of BTS_OFF_USR and BTS_OFF_OS in MSR_DEBUGCTLA is shown in 
Figure 17-12. The BTS_OFF_USR and BTS_OFF_OS fields may be implemented on other model- 
specific debug control register at different locations. 


See Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4 for a detailed description of each of the last branch recording MSRs. 


17.13.2 LBR Stack for Processors Based on Intel NetBurst® Microarchitecture 


The LBR stack is made up of LBR MSRs that are treated by the processor as a circular stack. The TOS pointer 
(MSR_LASTBRANCH_TOS MSR) points to the LBR MSR (or LBR MSR pair) that contains the most recent (last) 
branch record placed on the stack. Prior to placing a new branch record on the stack, the TOS is incremented by 1. 
When the TOS pointer reaches it maximum value, it wraps around to 0. See Table 17-17 and Figure 17-12. 


Table 17-17. LBR MSR Stack Size and TOS Pointer Range for the Pentium’ 4 and the Intel Xeon’ Processor Family 


DisplayFamily_DisplayModel Size of LBR Stack Range of TOS Pointer 
Family OFH, Models OH-02H; MSRs at locations 1DBH-1DEH. 4 Oto3 

Family OFH, Models; MSRs at locations 680H-68FH. 16 Oto15 

Family OFH, Model 03H; MSRs at locations 6COH-6CFH. 16 Oto15 


The registers in the LBR MSR stack and the MSR_LASTBRANCH_TOS MSR are read-only and can be read using the 
RDMSR instruction. 


Figure 17-13 shows the layout of a branch record in an LBR MSR (or MSR pair). Each branch record consists of two 
linear addresses, which represent the “from” and “to” instruction pointers for a branch, interrupt, or exception. The 
contents of the from and to addresses differ, depending on the source of the branch: 


° Taken branch — If the record is for a taken branch, the “from” address is the address of the branch instruction 
and the “to” address is the target instruction of the branch. 


¢ Interrupt — If the record is for an interrupt, the “from” address the return instruction pointer (RIP) saved for 
the interrupt and the “to” address is the address of the first instruction in the interrupt handler routine. The RIP 
is the linear address of the next instruction to be executed upon returning from the interrupt handler. 


¢ Exception — If the record is for an exception, the “from” address is the linear address of the instruction that 
caused the exception to be generated and the “to” address is the address of the first instruction in the 
exception handler routine. 
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CPUID Family 0FH, Models 0H-02H 
MSR_LASTBRANCH_0 through MSR_LASTBRANCH_3 
63 32 - 31 0 


To Linear Address From Linear Address 


CPUID Family 0FH, Model 03H-04H 
MSR_LASTBRANCH_0_FROM_IP through MSR_LASTBRANCH_15_FROM_IP 


63 32 - 31 0 


Reserved From Linear Address 


MSR_LASTBRANCH_0_TO_IP through MSR_LASTBRANCH_15_TO_IP 
63 32 - 31 0 


Reserved To Linear Address 


Figure 17-13. LBR MSR Branch Record Layout for the Pentium 4 
and Intel Xeon Processor Family 


Additional information is saved if an exception or interrupt occurs in conjunction with a branch instruction. If a 
branch instruction generates a trap type exception, two branch records are stored in the LBR stack: a branch record 
for the branch instruction followed by a branch record for the exception. 


If a branch instruction is immediately followed by an interrupt, a branch record is stored in the LBR stack for the 
branch instruction followed by a record for the interrupt. 


17.13.3. Last Exception Records 


The Pentium 4, Intel Xeon, Pentium M, Intel® Core™ Solo, Intel® Core™ Duo, Intel® Core™2 Duo, Intel® Core™ i7 
and Intel® Atom™ processors provide two MSRs (the MSR_LER_TO_LIP and the MSR_LER_FROM_LIP MSRs) that 
duplicate the functions of the LastExceptionToIP and LastExceptionFromIP MSRs found in the P6 family processors. 
The MSR_LER_TO_LIP and MSR_LER_FROM_LIP MSRs contain a branch record for the last branch that the 
processor took prior to an exception or interrupt being generated. 


17.14 LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING (INTEL® CORE™ 
SOLO AND INTEL® CORE™ DUO PROCESSORS) 


Intel Core Solo and Intel Core Duo processors provide last branch interrupt and exception recording. This capability 
is almost identical to that found in Pentium 4 and Intel Xeon processors. There are differences in the stack and in 
some MSR names and locations. 


Note the following: 


° IA32_DEBUGCTL MSR — Enables debug trace interrupt, debug trace store, trace messages enable, 
performance monitoring breakpoint flags, single stepping on branches, and last branch. IA32_DEBUGCTL MSR 
is located at register address 01D9H. 


See Figure 17-14 for the layout and the entries below for a description of the flags: 


— LBR (last branch/interrupt/exception) flag (bit 0) — When set, the processor records a running trace 
of the most recent branches, interrupts, and/or exceptions taken by the processor (prior to a debug 
exception being generated) in the last branch record (LBR) stack. For more information, see the “Last 
Branch Record (LBR) Stack” below. 


— BTF (single-step on branches) flag (bit 1) — When set, the processor treats the TF flag in the EFLAGS 
register as a “single-step on branches” flag rather than a “single-step on instructions” flag. This mechanism 
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allows single-stepping the processor on taken branches. See Section 17.4.3, “Single-Stepping on 
Branches,” for more information about the BTF flag. 


— TR (trace message enable) flag (bit 6) — When set, branch trace messages are enabled. When the 
processor detects a taken branch, interrupt, or exception; it sends the branch record out on the system bus 
as a branch trace message (BTM). See Section 17.4.4, “Branch Trace Messages,” for more information 
about the TR flag. 


— BTS (branch trace store) flag (bit 7) — When set, the flag enables BTS facilities to log BTMs to a 
memory-resident BTS buffer that is part of the DS save area. See Section 17.4.9, “BTS and DS Save Area.” 


— BTINT (branch trace interrupt) flag (bits 8) — When set, the BTS facilities generate an interrupt when 
the BTS buffer is full. When clear, BTMs are logged to the BTS buffer in a circular fashion. See Section 17.4.5, 
“Branch Trace Store (BTS),” for a description of this mechanism. 


31 876543210 


Reserved 


BTINT — Branch trace interrupt. 
BTS — Branch trace store 
TR — Trace messages enable 
Reserved 
BTF — Single-step on branches 
LBR — Last branch/interrupt/exception 


Figure 17-14. [A32_DEBUGCTL MSR for Intel Core Solo 
and Intel Core Duo Processors 


* Debug store (DS) feature flag (bit 21), returned by the CPUID instruction — Indicates that the 
processor provides the debug store (DS) mechanism, which allows BTMs to be stored in a memory-resident 
BTS buffer. See Section 17.4.5, “Branch Trace Store (BTS).” 


¢ Last Branch Record (LBR) Stack — The LBR stack consists of 8 MSRs (MSR_LASTBRANCH_O through 
MSR_LASTBRANCH_7); bits 31-0 hold the ‘from’ address, bits 63-32 hold the ‘to’ address (MSR addresses start 
at 40H). See Figure 17-15. 


¢ Last Branch Record Top-of-Stack (TOS) Pointer — The TOS Pointer MSR contains a 3-bit pointer (bits 2- 
0) to the MSR in the LBR stack that contains the most recent branch, interrupt, or exception recorded. For Intel 
Core Solo and Intel Core Duo processors, this MSR is located at register address 01C9H. 


For compatibility, the Intel Core Solo and Intel Core Duo processors provide two 32-bit MSRs (the 
MSR_LER_TO_LIP and the MSR_LER_FROM_LIP MSRs) that duplicate functions of the LastExceptionToIP and Last- 
ExceptionFromIP MSRs found in P6 family processors. 


For details, see Section 17.12, “Last Branch, Call Stack, Interrupt, and Exception Recording for Processors based 
on Skylake Microarchitecture,” and Section 2.20, “MSRs In Intel® Core™ Solo and Intel® Core™ Duo Processors” in 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 


MSR_LASTBRANCH_0 through MSR_LASTBRANCH_7 


63 32 - 31 0 
To Linear Address From Linear Address 


Figure 17-15. LBR Branch Record Layout for the Intel Core Solo 
and Intel Core Duo Processor 
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17.15 LAST BRANCH, INTERRUPT, AND EXCEPTION RECORDING (PENTIUM M 
PROCESSORS) 


Like the Pentium 4 and Intel Xeon processor family, Pentium M processors provide last branch interrupt and excep- 
tion recording. The capability operates almost identically to that found in Pentium 4 and Intel Xeon processors. 
There are differences in the shape of the stack and in some MSR names and locations. Note the following: 


¢ MSR_DEBUGCTLB MSR — Enables debug trace interrupt, debug trace store, trace messages enable, 
performance monitoring breakpoint flags, single stepping on branches, and last branch. For Pentium M 
processors, this MSR is located at register address 01D9H. See Figure 17-16 and the entries below for a 
description of the flags. 


— LBR (last branch/interrupt/exception) flag (bit 0) — When set, the processor records a running trace 
of the most recent branches, interrupts, and/or exceptions taken by the processor (prior to a debug 
exception being generated) in the last branch record (LBR) stack. For more information, see the “Last 
Branch Record (LBR) Stack” bullet below. 


— BTF (single-step on branches) flag (bit 1) — When set, the processor treats the TF flag in the EFLAGS 
register as a “single-step on branches” flag rather than a “single-step on instructions” flag. This mechanism 
allows single-stepping the processor on taken branches. See Section 17.4.3, “Single-Stepping on 
Branches,” for more information about the BTF flag. 


— PBi (performance monitoring/breakpoint pins) flags (bits 5-2) — When these flags are set, the 
performance monitoring/breakpoint pins on the processor (BP0#, BP1#, BP2#, and BP3#) report 
breakpoint matches in the corresponding breakpoint-address registers (DRO through DR3). The processor 
asserts then deasserts the corresponding BP/# pin when a breakpoint match occurs. When a PBi flag is 
clear, the performance monitoring/breakpoint pins report performance events. Processor execution is not 
affected by reporting performance events. 


— TR (trace message enable) flag (bit 6) — When set, branch trace messages are enabled. When the 
processor detects a taken branch, interrupt, or exception, it sends the branch record out on the system bus 
as a branch trace message (BTM). See Section 17.4.4, “Branch Trace Messages,” for more information 
about the TR flag. 


— BTS (branch trace store) flag (bit 7) — When set, enables the BTS facilities to log BTMs to a memory- 
resident BTS buffer that is part of the DS save area. See Section 17.4.9, “BTS and DS Save Area.” 


— BTINT (branch trace interrupt) flag (bits 8) — When set, the BTS facilities generate an interrupt when 
the BTS buffer is full. When clear, BTMs are logged to the BTS buffer in a circular fashion. See Section 17.4.5, 
“Branch Trace Store (BTS),” for a description of this mechanism. 


31 876543210 


Reserved 


BTINT — Branch trace interrupt 
BTS — Branch trace store 
TR — Trace messages enable 
PB3/2/1/0 — Performance monitoring breakpoint flags 
BTF — Single-step on branches 
LBR — Last branch/interrupt/exception 


Figure 17-16. MSR_DEBUGCTLB MSR for Pentium M Processors 


* Debug store (DS) feature flag (bit 21), returned by the CPUID instruction — Indicates that the 
processor provides the debug store (DS) mechanism, which allows BTMs to be stored in a memory-resident 
BTS buffer. See Section 17.4.5, “Branch Trace Store (BTS).” 
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¢ Last Branch Record (LBR) Stack — The LBR stack consists of 8 MSRs (MSR_LASTBRANCH_O through 
MSR_LASTBRANCH_7); bits 31-0 hold the ‘from’ address, bits 63-32 hold the ‘to’ address. For Pentium M 
Processors, these pairs are located at register addresses 040H-047H. See Figure 17-17. 


¢ Last Branch Record Top-of-Stack (TOS) Pointer — The TOS Pointer MSR contains a 3-bit pointer (bits 2- 
0) to the MSR in the LBR stack that contains the most recent branch, interrupt, or exception recorded. For 
Pentium M Processors, this MSR is located at register address 01C9H. 


MSR_LASTBRANCH_0 _ through MSR_LASTBRANCH_7 


63 32 - 31 0 
To Linear Address From Linear Address 


Figure 17-17. LBR Branch Record Layout for the Pentium M Processor 


For more detail on these capabilities, see Section 17.13.3, “Last Exception Records,” and Section 2.21, “MSRs In 
the Pentium M Processor” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 


17.16 LAST BRANCH, INTERRUPT, AND EXCEPTION 
RECORDING (P6 FAMILY PROCESSORS) 


The P6 family processors provide five MSRs for recording the last branch, interrupt, or exception taken by the 
processor: DEBUGCTLMSR, LastBranchToIP, LastBranchFromIP, LastExceptionToIP, and LastExceptionFromIP. 
These registers can be used to collect last branch records, to set breakpoints on branches, interrupts, and excep- 
tions, and to single-step from one branch to the next. 


See Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4 for a detailed description of each of the last branch recording MSRs. 


17.16.1 DEBUGCTLMSR Register 


The version of the DEBUGCTLMSR register found in the P6 family processors enables last branch, interrupt, and 
exception recording; taken branch breakpoints; the breakpoint reporting pins; and trace messages. This register 
can be written to using the WRMSR instruction, when operating at privilege level 0 or when in real-address mode. 
A protected-mode operating system procedure is required to provide user access to this register. Figure 17-18 
shows the flags in the DEBUGCTLMSR register for the P6 family processors. The functions of these flags are as 
follows: 


¢ LBR (last branch/interrupt/exception) flag (bit 0) — When set, the processor records the source and 
target addresses (in the LastBranchToIP, LastBranchFromIP, LastExceptionToIP, and LastExceptionFromIP 
MSRs) for the last branch and the last exception or interrupt taken by the processor prior to a debug exception 
being generated. The processor clears this flag whenever a debug exception, such as an instruction or data 
breakpoint or single-step trap occurs. 
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31 76543210 


_|P|P|P/P|B 
Reserved B|B/B/B|T 
R} 3) 2/1/o]F 


DOr 


TR — Trace messages enable 
PBi — Performance monitoring/breakpoint pins 
BTF — Single-step on branches 
LBR — Last branch/interrupt/exception 


Figure 17-18. DEBUGCTLMSR Register (P6 Family Processors) 


° BTF (single-step on branches) flag (bit 1) — When set, the processor treats the TF flag in the EFLAGS 
register as a “single-step on branches” flag. See Section 17.4.3, “Single-Stepping on Branches.” 


° PBi (performance monitoring/breakpoint pins) flags (bits 2 through 5) — When these flags are set, 
the performance monitoring/breakpoint pins on the processor (BPO#, BP1#, BP2#, and BP3#) report 
breakpoint matches in the corresponding breakpoint-address registers (DRO through DR3). The processor 
asserts then deasserts the corresponding BP/# pin when a breakpoint match occurs. When a PBi flag is clear, 
the performance monitoring/breakpoint pins report performance events. Processor execution is not affected by 
reporting performance events. 


° TR (trace message enable) flag (bit 6) — When set, trace messages are enabled as described in Section 
17.4.4, “Branch Trace Messages.” Setting this flag greatly reduces the performance of the processor. When 
trace messages are enabled, the values stored in the LastBranchToIP, LastBranchFromIP, LastExceptionToIP, 
and LastExceptionFromIP MSRs are undefined. 


17.16.2 Last Branch and Last Exception MSRs 


The LastBranchToIP and LastBranchFromIP MSRs are 32-bit registers for recording the instruction pointers for the 
last branch, interrupt, or exception that the processor took prior to a debug exception being generated. When a 
branch occurs, the processor loads the address of the branch instruction into the LastBranchFromIP MSR and loads 
the target address for the branch into the LastBranchToIP MSR. 


When an interrupt or exception occurs (other than a debug exception), the address of the instruction that was 
interrupted by the exception or interrupt is loaded into the LastBranchFromIP MSR and the address of the exception 
or interrupt handler that is called is loaded into the LastBranchToIP MSR. 


The LastExceptionToIP and LastExceptionFromIP MSRs (also 32-bit registers) record the instruction pointers for the 
last branch that the processor took prior to an exception or interrupt being generated. When an exception or inter- 
rupt occurs, the contents of the LastBranchToIP and LastBranchFromIP MSRs are copied into these registers before 
the to and from addresses of the exception or interrupt are recorded in the LastBranchToIP and LastBranchFromIP 
MSRs. 


These registers can be read using the RDMSR instruction. 


Note that the values stored in the LastBranchToIP, LastBranchFromIP, LastExceptionToIP, and LastExceptionFromIP 
MSRs are offsets into the current code segment, as opposed to linear addresses, which are saved in last branch 
records for the Pentium 4 and Intel Xeon processors. 


17.16.3 Monitoring Branches, Exceptions, and Interrupts 


When the LBR flag in the DEBUGCTLMSR register is set, the processor automatically begins recording branches that 
it takes, exceptions that are generated (except for debug exceptions), and interrupts that are serviced. Each time 
a branch, exception, or interrupt occurs, the processor records the to and from instruction pointers in the Last- 
BranchToIP and LastBranchFromIP MSRs. In addition, for interrupts and exceptions, the processor copies the 
contents of the LastBranchToIP and LastBranchFromIP MSRs into the LastExceptionToIP and LastExceptionFromIP 
MSRs prior to recording the to and from addresses of the interrupt or exception. 
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When the processor generates a debug exception (#DB), it automatically clears the LBR flag before executing the 
exception handler, but does not touch the last branch and last exception MSRs. The addresses for the last branch, 
interrupt, or exception taken are thus retained in the LastBranchToIP and LastBranchFromIP MSRs and the 
addresses of the last branch prior to an interrupt or exception are retained in the LastExceptionToIP, and LastEx- 
ceptionFromIP MSRs. 


The debugger can use the last branch, interrupt, and/or exception addresses in combination with code-segment 
selectors retrieved from the stack to reset breakpoints in the breakpoint-address registers (DRO through DR3), 
allowing a backward trace from the manifestation of a particular bug toward its source. Because the instruction 
pointers recorded in the LastBranchToIP, LastBranchFromIP, LastExceptionToIP, and LastExceptionFromIP MSRs 
are offsets into a code segment, software must determine the segment base address of the code segment associ- 
ated with the control transfer to calculate the linear address to be placed in the breakpoint-address registers. The 
segment base address can be determined by reading the segment selector for the code segment from the stack 
and using it to locate the segment descriptor for the segment in the GDT or LDT. The segment base address can 
then be read from the segment descriptor. 


Before resuming program execution from a debug-exception handler, the handler must set the LBR flag again to re- 
enable last branch and last exception/interrupt recording. 


17.17. TIME-STAMP COUNTER 


The Intel 64 and IA-32 architectures (beginning with the Pentium processor) define a time-stamp counter mecha- 
nism that can be used to monitor and identify the relative time occurrence of processor events. The counter’s archi- 
tecture includes the following components: 


° TSC flag — A feature bit that indicates the availability of the time-stamp counter. The counter is available in an 
if the function CPUID.1:EDX.TSC[bit 4] = 1. 


° IA32_TIME_STAMP_COUNTER MSR (called TSC MSR in P6 family and Pentium processors) — The MSR used 
as the counter. 


* RDTSC instruction — An instruction used to read the time-stamp counter. 


¢ TSD flag — A control register flag is used to enable or disable the time-stamp counter (enabled if 
CR4.TSD[bit 2] = 1). 


The time-stamp counter (as implemented in the P6 family, Pentium, Pentium M, Pentium 4, Intel Xeon, Intel Core 
Solo and Intel Core Duo processors and later processors) is a 64-bit counter that is set to 0 following a RESET of 
the processor. Following a RESET, the counter increments even when the processor is halted by the HLT instruction 
or the external STPCLK# pin. Note that the assertion of the external DPSLP# pin may cause the time-stamp 
counter to stop. 


Processor families increment the time-stamp counter differently: 


¢ For Pentium M processors (family [06H], models [09H, ODH]); for Pentium 4 processors, Intel Xeon processors 
(family [OFH], models [00H, 01H, or 02H]); and for P6 family processors: the time-stamp counter increments 
with every internal processor clock cycle. 


The internal processor clock cycle is determined by the current core-clock to bus-clock ratio. Intel® 
SpeedStep® technology transitions may also impact the processor clock. 


¢ For Pentium 4 processors, Intel Xeon processors (family [OFH], models [03H and higher]); for Intel Core Solo 
and Intel Core Duo processors (family [06H], model [OEH]); for the Intel Xeon processor 5100 series and Intel 
Core 2 Duo processors (family [06H], model [OFH]); for Intel Core 2 and Intel Xeon processors (family [06H], 
DisplayModel [17H]); for Intel Atom processors (family [06H], 
DisplayModel [1CH]): the time-stamp counter increments at a constant rate. That rate may be set by the 
maximum core-clock to bus-clock ratio of the processor or may be set by the maximum resolved frequency at 
which the processor is booted. The maximum resolved frequency may differ from the processor base 
frequency, see Section 18.7.2 for more detail. On certain processors, the TSC frequency may not be the same 
as the frequency in the brand string. 


The specific processor configuration determines the behavior. Constant TSC behavior ensures that the duration 
of each clock tick is uniform and supports the use of the TSC as a wall clock timer even if the processor core 
changes frequency. This is the architectural behavior moving forward. 
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NOTE 


To determine average processor clock frequency, Intel recommends the use of performance 
monitoring logic to count processor core clocks over the period of time for which the average is 
required. See Section 18.6.4.5, “Counting Clocks on systems with Intel Hyper-Threading 
Technology in Processors Based on Intel NetBurst® Microarchitecture,” and https://perfmon- 
events.intel.com/ for more information. 


The RDTSC instruction reads the time-stamp counter and is guaranteed to return a monotonically increasing 
unique value whenever executed, except for a 64-bit counter wraparound. Intel guarantees that the time-stamp 
counter will not wraparound within 10 years after being reset. The period for counter wrap is longer for Pentium 4, 
Intel Xeon, P6 family, and Pentium processors. 


Normally, the RDTSC instruction can be executed by programs and procedures running at any privilege level and in 
virtual-8086 mode. The TSD flag allows use of this instruction to be restricted to programs and procedures running 
at privilege level 0. A secure operating system would set the TSD flag during system initialization to disable user 
access to the time-stamp counter. An operating system that disables user access to the time-stamp counter should 
emulate the instruction through a user-accessible programming interface. 


The RDTSC instruction is not serializing or ordered with other instructions. It does not necessarily wait until all 
previous instructions have been executed before reading the counter. Similarly, subsequent instructions may begin 
execution before the RDTSC instruction operation is performed. 


The RDMSR and WRMSR instructions read and write the time-stamp counter, treating the time-stamp counter as an 
ordinary MSR (address 10H). In the Pentium 4, Intel Xeon, and P6 family processors, all 64-bits of the time-stamp 
counter are read using RDMSR (just as with RDTSC). When WRMSR is used to write the time-stamp counter on 
processors before family [OFH], models [03H, 04H]: only the low-order 32-bits of the time-stamp counter can be 
written (the high-order 32 bits are cleared to 0). For family [OFH], models [03H, 04H, 06H]; for family [06H]], 
model [OEH, OFH]; for family [06H]], DisplayModel [17H, 1AH, 1CH, 1DH]: all 64 bits are writable. 


17.17.1 Invariant TSC 


The time stamp counter in newer processors may support an enhancement, referred to as invariant TSC. 
Processor’s support for invariant TSC is indicated by CPUID.80000007H:EDX[8]. 


The invariant TSC will run at a constant rate in all ACPI P-, C-. and T-states. This is the architectural behavior 
moving forward. On processors with invariant TSC support, the OS may use the TSC for wall clock timer services 
(instead of ACPI or HPET timers). TSC reads are much more efficient and do not incur the overhead associated with 
a ring transition or access to a platform resource. 


17.17.2 IA32_TSC_AUX Register and RDTSCP Support 


Processors based on Intel microarchitecture code name Nehalem provide an auxiliary TSC register, IA32_TSC_AUX 
that is designed to be used in conjunction with IA32_TSC. IA32_TSC_AUX provides a 32-bit field that is initialized 
by privileged software with a signature value (for example, a logical processor ID). 


The primary usage of IA32_TSC_AUX in conjunction with IA32_TSC is to allow software to read the 64-bit time 
stamp in IA32_TSC and signature value in IA32_TSC_AUX with the instruction RDTSCP in an atomic operation. 
RDTSCP returns the 64-bit time stamp in EDX:EAX and the 32-bit TSC_AUX signature value in ECX. The atomicity 
of RDTSCP ensures that no context switch can occur between the reads of the TSC and TSC_AUX values. 


Support for RDTSCP is indicated by CPUID.80000001H:EDX[27]. As with RDTSC instruction, non-ring 0 access is 
controlled by CR4.TSD (Time Stamp Disable flag). 


User mode software can use RDTSCP to detect if CPU migration has occurred between successive reads of the TSC. 
It can also be used to adjust for per-CPU differences in TSC values in a NUMA system. 
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17.17.3. Time-Stamp Counter Adjustment 


Software can modify the value of the time-stamp counter (TSC) of a logical processor by using the WRMSR instruc- 
tion to write to the IA32_TIME_STAMP_COUNTER MSR (address 10H). Because such a write applies only to that 
logical processor, software seeking to synchronize the TSC values of multiple logical processors must perform these 
writes on each logical processor. It may be difficult for software to do this in a way than ensures that all logical 
processors will have the same value for the TSC at a given point in time. 


The synchronization of TSC adjustment can be simplified by using the 64-bit IA32_TSC_ADJUST MSR (address 
3BH). Like the IA32_TIME_STAMP_COUNTER MSR, the IA32_TSC_ADJUST MSR is maintained separately for each 
logical processor. A logical processor maintains and uses the IA32_TSC_ADJUST MSR as follows: 

® On RESET, the value of the IA32_TSC_ADJUST MSR is 0. 


¢ If an execution of WRMSR to the IA32_TIME_STAMP_COUNTER MSR adds (or subtracts) value X from the TSC, 
the logical processor also adds (or subtracts) value X from the IA32_TSC_ADJUST MSR. 


° If an execution of WRMSR to the IA32_TSC_ADJUST MSR adds (or subtracts) value X from that MSR, the logical 
processor also adds (or subtracts) value X from the TSC. 


Unlike the TSC, the value of the IA32_TSC_ADJUST MSR changes only in response to WRMSR (either to the MSR 
itself, or to the IA32_TIME_STAMP_COUNTER MSR). Its value does not otherwise change as time elapses. Software 
seeking to adjust the TSC can do so by using WRMSR to write the same value to the IA32_TSC_ADJUST MSR on 
each logical processor. 


Processor support for the IA32_TSC_ADJUST MSR is indicated by CPUID.(EAX=07H, ECX=0H):EBX.TSC_ADJUST 
(bit 1). 


17.17.4 Invariant Time-Keeping 


The invariant TSC is based on the invariant timekeeping hardware (called Always Running Timer or ART), that runs 
at the core crystal clock frequency. The ratio defined by CPUID leaf 15H expresses the frequency relationship 
between the ART hardware and TSC. 


If CPUID.15H:EBX[31:0] != 0 and CPUID.80000007H:EDX[InvariantTSC] = 1, the following linearity relationship 
holds between TSC and the ART hardware: 


TSC_Value = (ART_Value * CPUID.15H:EBX[31:0] )/ CPUID.15H:EAX[31:0] + K 
Where 'K' is an offset that can be adjusted by a privileged agent?. 
When ART hardware is reset, both invariant TSC and K are also reset. 


17.18 INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) MONITORING 
FEATURES 


The Intel Resource Director Technology (Intel RDT) feature set provides a set of monitoring capabilities including 
Cache Monitoring Technology (CMT) and Memory Bandwidth Monitoring (MBM). The Intel® Xeon® processor E5 v3 
family introduced resource monitoring capability in each logical processor to measure specific platform shared 
resource metrics, for example, L3 cache occupancy. The programming interface for these monitoring features is 
described in this section. Two features within the monitoring feature set provided are described - Cache Monitoring 
Technology (CMT) and Memory Bandwidth Monitoring. 


Cache Monitoring Technology (CMT) allows an Operating System, Hypervisor or similar system management agent 
to determine the usage of cache by applications running on the platform. The initial implementation is directed at 
L3 cache monitoring (currently the last level cache in most server platforms). 


Memory Bandwidth Monitoring (MBM), introduced in the Intel® Xeon® processor E5 v4 family, builds on the CMT 
infrastructure to allow monitoring of bandwidth from one level of the cache hierarchy to the next - in this case 


2. IA32_TSC_ADJUST MSR and the TSC-offset field in the VM execution controls of VMCS are some of the common interfaces that priv- 
ileged software can use to manage the time stamp counter for keeping time 
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focusing on the L3 cache, which is typically backed directly by system memory. As a result of this implementation, 
memory bandwidth can be monitored. 


The monitoring mechanisms described provide the following key shared infrastructure features: 


¢ Amechanism to enumerate the presence of the monitoring capabilities within the platform (via a CPUID feature 
bit). 


¢ A framework to enumerate the details of each sub-feature (including CMT and MBM, as discussed later, via 
CPUID leaves and sub-leaves). 


¢ Amechanism for the OS or Hypervisor to indicate a software-defined ID for each of the software threads (appli- 
cations, virtual machines, etc.) that are scheduled to run on a logical processor. These identifiers are known as 
Resource Monitoring IDs (RMIDs). 


® Mechanisms in hardware to monitor cache occupancy and bandwidth statistics as applicable to a given product 
generation on a per software-id basis. 


® Mechanisms for the OS or Hypervisor to read back the collected metrics such as L3 occupancy or Memory 
Bandwidth for a given software ID at any point during runtime. 


17.18.1 Overview of Cache Monitoring Technology and Memory Bandwidth Monitoring 


The shared resource monitoring features described in this chapter provide a layer of abstraction between applica- 
tions and logical processors through the use of Resource Monitoring IDs (RMIDs). Each logical processor in the 
system can be assigned an RMID independently, or multiple logical processors can be assigned to the same RMID 
value (e.g., to track an application with multiple threads). For each logical processor, only one RMID value is active 
at a time. This is enforced by the IA32_PQR_ASSOC MSR, which specifies the active RMID of a logical processor. 

Writing to this MSR by software changes the active RMID of the logical processor from an old value to a new value. 


The underlying platform shared resource monitoring hardware tracks cache metrics such as cache utilization and 
misses as a result of memory accesses according to the RMIDs and reports monitored data via a counter register 
(IA32_QM_CTR). The specific event types supported vary by generation and can be enumerated via CPUID. Before 
reading back monitored data software must configure an event selection MSR (IA32_QM_EVTSEL) to specify which 
metric is to be reported, and the specific RMID for which the data should be returned. 


Processor support of the monitoring framework and sub-features such as CMT is reported via the CPUID instruc- 
tion. The resource type available to the monitoring framework is enumerated via a new leaf function in CPUID. 
Reading and writing to the monitoring MSRs requires the RDMSR and WRMSR instructions. 


The Cache Monitoring Technology feature set provides the following unique mechanisms: 


¢ Amechanism to enumerate the presence and details of the CMT feature as applicable to a given level of the 
cache hierarchy, independent of other monitoring features. 


° CMT-specific event codes to read occupancy for a given level of the cache hierarchy. 
The Memory Bandwidth Monitoring feature provides the following unique mechanisms: 


¢ Amechanism to enumerate the presence and details of the MBM feature as applicable to a given level of the 
cache hierarchy, independent of other monitoring features. 


¢ MBM-specific event codes to read bandwidth out to the next level of the hierarchy and various sub-event codes 
to read more specific metrics as discussed later (e.g., total bandwidth vs. bandwidth only from local memory 
controllers on the same package). 


17.18.2 Enabling Monitoring: Usage Flow 


Figure 17-19 illustrates the key steps for OS/VMM to detect support of shared resource monitoring features such as 
CMT and enable resource monitoring for available resource types and monitoring events. 
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On OS/VMM Initialization On Context Switch Periodical Resource 
Selection/Reporting 
PQM Capability Set RMID to monitor Configure event type 
Enumeration the scheduled app Read monitored data 
CPUID[ WRMSR RDMSR/WRMSR 

CPUID.(7,0):EBX.12 1A32_PQR_ASSOC.RMID 1A32 QM EVTSEL 
CPUID.(OFH,0):EDX[31:1] lita 
CPUID.(OFH,1):ECX([31:0] 1A32_QM_CTR 
CPUID.(OFH,1):EDX[31:0] 
CPUID.(OFH,1):EBX([31:0] 
CPUID.(OFH,0):EBX[31:0] 


Figure 17-19. Platform Shared Resource Monitoring Usage Flow 


17.18.3 Enumeration and Detecting Support of Cache Monitoring Technology and Memory 


Bandwidth Monitoring 


Software can query processor support of shared resource monitoring features capabilities by executing CPUID 
instruction with EAX = 07H, ECX = OH as input. If CPUID.(EAX=07H, ECX=0):EBX.PQM[bit 12] reports 1, the 
processor provides the following programming interfaces for shared resource monitoring, including Cache Moni- 
toring Technology: 


CPUID leaf function OFH (Shared Resource Monitoring Enumeration leaf) provides information on available 
resource types (see Section 17.18.4), and monitoring capabilities for each resource type (see Section 
17.18.5). Note CMT and MBM capabilities are enumerated as separate event vectors using shared enumeration 
infrastructure under a given resource type. 


IA32_PQR_ASSOC.RMID: The per-logical-processor MSR, IA32_PQR_ASSOC, that OS/VMM can use to assign 
an RMID to each logical processor, see Section 17.18.6. 

IA32_QM_EVTSEL: This MSR specifies an Event ID (EvtID) and an RMID which the platform uses to look up and 
provide monitoring data in the monitoring counter, IA32_QM_CTR, see Section 17.18.7. 


IA32_QM_CTR: This MSR reports monitored resource data when available along with bits to allow software to 
check for error conditions and verify data validity. 


Software must follow the following sequence of enumeration to discover Cache Monitoring Technology capabilities: 


1. 
2. 


Execute CPUID with EAX=0 to discover the “cpuid_maxLeaf” supported in the processor; 

If cpuid_maxLeaf >= 7, then execute CPUID with EAX=7, ECX= 0 to verify CPUID.(EAX=07H, 
ECX=0):EBX.PQM[bit 12] is set; 

If CPUID.(EAX=07H, ECX=0):EBX.PQM[bit 12] = 1, then execute CPUID with EAX=OFH, ECX= 0 to query 
available resource types that support monitoring; 


If CPUID.(EAX=0FH, ECX=0):EDX.L3[bit 1] = 1, then execute CPUID with EAX=0FH, ECX= 1 to query the 
specific capabilities of L3 Cache Monitoring Technology (CMT) and Memory Bandwidth Monitoring. 


If CPUID.(EAX=OFH, ECX=0):EDX reports additional resource types supporting monitoring, then execute 
CPUID with EAX=0FH, ECX set to a corresponding resource type ID (ResID) as enumerated by the bit position 
of CPUID.(EAX=0FH, ECX=0):EDX. 


17.18.4 Monitoring Resource Type and Capability Enumeration 


CPUID leaf function OFH (Shared Resource Monitoring Enumeration leaf) provides one sub-leaf (sub-function 0) 
that reports shared enumeration infrastructure, and one or more sub-functions that report feature-specific 
enumeration data: 


Monitoring leaf sub-function 0 enumerates available resources that support monitoring, i.e. executing CPUID 
with EAX=0FH and ECX=OH. In the initial implementation, L3 cache is the only resource type available. Each 
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supported resource type is represented by a bit in CPUID.(EAX=0FH, ECX=0):EDX[31:1]. The bit position 
corresponds to the sub-leaf index (ResID) that software must use to query details of the monitoring capability 
of that resource type (see Figure 17-21 and Figure 17-22). Reserved bits of CPUID.(EAX=0FH, 
ECX=0):EDX[31:2] correspond to unsupported sub-leaves of the CPUID.OFH leaf. Additionally, 
CPUID.(EAX=0FH, ECX=0H):EBX reports the highest RMID value of any resource type that supports monitoring 
in the processor. 


CPUID.(EAX=0FH, ECX=0H) Output: (EAX: Reserved; ECX: Reserved) 


31 2 


woOrfla 


EDX 


Reserved 


31 0 
Highest RMID Value of Any Resource Type (Zero-Based) 


EBX 


Figure 17-20. CPUID.(EAX=OFH, ECX=0H) Monitoring Resource Type Enumeration 


17.18.5 Feature-Specific Enumeration 


Each additional sub-leaf of CPUID.(EAX=OFH, ECX=ResID) enumerates the specific details for software to program 
Monitoring MSRs using the resource type associated with the given ResID. 


Note that in future Monitoring implementations the meanings of the returned registers may vary in other sub- 
leaves that are not yet defined. The registers will be specified and defined on a per-ResID basis. 


CPUID.(EAX=0FH, ECX=1H) Output: (EAX: Reserved) 


31 0 
EBX : s 
Upscaling Factor to Total Occupancy (Bytes) Upscaling Factor 
31 0 
ECX ; ; MaxRMID 
Highest RMID Value of This Resource Type (Zero-Based) x 


Figure 17-21. L3 Cache Monitoring Capability Enumeration Data (CPUID.(EAX=OFH, ECX=1H) ) 


For each supported Cache Monitoring resource type, hardware supports only a finite number of RMIDs. 
CPUID.(EAX=0FH, ECX=1H).ECX enumerates the highest RMID value that can be monitored with this resource 
type, see Figure 17-21. 

CPUID.(EAX=0FH, ECX=1H).EDX specifies a bit vector that is used to look up the EventID (See Figure 17-22 and 
Table 17-18) that software must program with IA32_QM_EVTSEL in order to retrieve event data. After software 
configures IA32_QMEVTSEL with the desired RMID and EventID, it can read the resulting data from IA32_QM_CTR. 
The raw numerical value reported from IA32_QM_CTR can be converted to the final value (occupancy in bytes or 
bandwidth in bytes per sampled time period) by multiplying the counter value by the value from CPUID.(EAX=0FH, 
ECX=1H).EBX, see Figure 17-21. 
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EventTypeBitMask 
31 3.21 0 


EDX 


Reserved | 


L3 Occupancy 
L3 Total BW 
L3 Local BW 


Figure 17-22. L3 Cache Monitoring Capability Enumeration Event Type Bit Vector (CPUID.(EAX=OFH, ECX=1H) ) 


17.18.5.1 Cache Monitoring Technology 


On processors for which Cache Monitoring Technology supports the L3 cache occupancy event, CPUID.(EAX=0FH, 
ECX=1H).EDX would return with only bit 0 set. The corresponding event ID can be looked up from Table 17-18. The 
L3 occupancy data accumulated in IA32_QM_CTR can be converted to total occupancy (in bytes) by multiplying 
with CPUID.(EAX=0FH, ECX=1H).EBX. 


Event codes for Cache Monitoring Technology are discussed in the next section. 


17.18.5.2 Memory Bandwidth Monitoring 


On processors that monitoring supports Memory Bandwidth Monitoring using ResID=1 (L3), two additional bits will 
be set in the vector at CPUID.(EAX=0FH, ECX=1H).EDX: 


¢ CPUID.(EAX=0FH, ECX=1H).EDX[bit 1]: indicates the L3 total external bandwidth monitoring event is 
supported if set. This event monitors the L3 total external bandwidth to the next level of the cache hierarchy, 
including all demand and prefetch misses from the L3 to the next hierarchy of the memory system. In most 
platforms, this represents memory bandwidth. 


¢ CPUID.(EAX=0FH, ECX=1H).EDX[bit 2]: indicates L3 local memory bandwidth monitoring event is supported if 
set. This event monitors the L3 external bandwidth satisfied by the local memory. In most platforms that 
support this event, L3 requests are likely serviced by a memory system with non-uniform memory archi- 
tecture. This allows bandwidth to off-package memory resources to be tracked by subtracting local from total 
bandwidth (for instance, bandwidth over QPI to a memory controller on another physical processor could be 
tracked by subtraction). 


The corresponding Event ID can be looked up from Table 17-18. The L3 bandwidth data accumulated in 
IA32_QM_CTR can be converted to total bandwidth (in bytes) using CPUID.(EAX=0FH, ECX=1H).EBX. 


Table 17-18. Monitoring Supported Event IDs 


Event Type Event ID Context 

L3 Cache Occupancy 01H Cache Monitoring Technology 
L3 Total External Bandwidth 02H MBM 

L3 Local External Bandwidth 03H MBM 

Reserved All other event codes N/A 


17.18.6 Monitoring Resource RMID Association 


After Monitoring and sub-features has been enumerated, software can begin using the monitoring features. The 
first step is to associate a given software thread (or multiple threads as part of an application, VM, group of appli- 
cations or other abstraction) with an RMID. 


Note that the process of associating an RMID with a given software thread is the same for all shared resource moni- 
toring features (CMT, MBM), and a given RMID number has the same meaning from the viewpoint of any logical 
processors in a package. Stated another way, a thread may be associated in a 1:1 mapping with an RMID, and that 
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RMID may allow cache occupancy, memory bandwidth information or other monitoring data to be read back later 
with monitoring event codes (retrieving data is discussed in a previous section). 


The association of an application thread with an RMID requires an OS to program the per-logical-processor MSR 
IA32_PQR_ASSOC at context swap time (updates may also be made at any other arbitrary points during program 
execution such as application phase changes). The IA32_PQR_ASSOC MSR specifies the active RMID that moni- 
toring hardware will use to tag internal operations, such as L3 cache requests. The layout of the MSR is shown in 
Figure 17-23. Software specifies the active RMID to monitor in the IA32_PQR_ASSOC.RMID field. The width of the 
RMID field can vary from one implementation to another, and is derived from Ceil (LOG> ( 1 + CPUID.(EAX=O0FH, 
ECX=0):EBX[31:0])). The value of IA32_PQR_ASSOC after power-on is 0. 


Width of IA32_PQR_ASSOC.RMID field: Logg ( CPUID.(EAX=0FH, ECX=0H).EBX[31:0] +1) 
63 32 31 10 9 0 
7 Reserved RMID 1A32_PQR_ASSOC 


Gs 
Z 
@ 


*See Section 17.18 


Figure 17-23. |A32_PQR_ASSOC MSR 


In the initial implementation, the width of the RMID field is up to 10 bits wide, zero-referenced and fully encoded. 
However, software must use CPUID to query the maximum RMID supported by the processor. If a value larger than 
the maximum RMID is written to IA32_PQR_ASSOC.RMID, a #GP(0) fault will be generated. 


RMIDs have a global scope within the physical package- if an RMID is assigned to one logical processor then the 
same RMID can be used to read multiple thread attributes later (for example, L3 cache occupancy or external 
bandwidth from the L3 to the next level of the cache hierarchy). In a multiple LLC platform the RMIDs are to be 
reassigned by the OS or VMM scheduler when an application is migrated across LLCs. 


Note that in a situation where Monitoring supports multiple resource types, some upper range of RMIDs (e.g. RMID 
31) may only be supported by one resource type but not by another resource type. 


17.18.7 Monitoring Resource Selection and Reporting Infrastructure 


The reporting mechanism for Cache Monitoring Technology and other related features is architecturally exposed as 
an MSR pair that can be programmed and read to measure various metrics such as the L3 cache occupancy (CMT) 
and bandwidths (MBM) depending on the level of Monitoring support provided by the platform. Data is reported 
back on a per-RMID basis. These events do not trigger based on event counts or trigger APIC interrupts (e.g. no 
Performance Monitoring Interrupt occurs based on counts). Rather, they are used to sample counts explicitly. 


The MSR pair for the shared resource monitoring features (CMT, MBM) is separate from and not shared with archi- 
tectural Perfmon counters, meaning software can use these monitoring features simultaneously with the Perfmon 
counters. 


Access to the aggregated monitoring information is accomplished through the following programmable monitoring 
MSRs: 


¢ TA32_QM_EVTSEL: This MSR provides a role similar to the event select MSRs for programmable performance 
monitoring described in Chapter 18. The simplified layout of the MSR is shown in Figure 17-24. Bits 
IA32_QM_EVTSEL.EvtID (bits 7:0) specify an event code of a supported resource type for hardware to report 
monitored data associated with IA32_QM_EVTSEL.RMID (bits 41:32). Software can configure 
IA32_QM_EVTSEL.RMID with any RMID that is active within the physical processor. The width of 
IA32_QM_EVTSEL.RMID matches that of IA32_PQR_ASSOC.RMID. Supported event codes for the 
IA32_QM_EVTSEL register are shown in Table 17-18. Note that valid event codes may not necessarily map 
directly to the bit position used to enumerate support for the resource via CPUID. 


Software can program an RMID / Event ID pair into the IA32_QM_EVTSEL MSR bit field to select an RMID to 
read a particular counter for a given resource. The currently supported list of Monitoring Event IDs is discussed 
in Section 17.18.5, which covers feature-specific details. 
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Thread access to the IA32_QM_EVTSEL and IA32_QM_CTR MSR pair should be serialized (that is, treated as a 
critical section under lock) to avoid situations where one thread changes the RMID/EvtID just before another 
thread reads monitoring data from IA32_QM_CTR. 


¢ TA32_QM_CTR: This MSR reports monitored data when available. It contains three bit fields. If software 
configures an unsupported RMID or event type in IA32_QM_EVTSEL, then IA32_QM_CTR.Error (bit 63) will be 
set, indicating there is no valid data to report. If IA32_QM_CTR.Unavailable (bit 62) is set, it indicates 
monitored data for the RMID is not available, and IA32_QM_CTR.data (bits 61:0) should be ignored. Therefore, 
IA32_QM_CTR.data (bits 61:0) is valid only if bit 63 and 62 are both clear. For Cache Monitoring Technology, 
software can convert IA32_QM_CTR.data into cache occupancy or bandwidth metrics expressed in bytes by 
multiplying with the conversion factor from CPUID.(EAX=0FH, ECX=1H).EBX. 


63 4241 32 31 87 0 
Reserved | RMID Reserved EvtID 1A32_QM_EVTSEL 
63. «61 0 
E | u Resource Monitoring Data 1A32_QM_CTR 


Figure 17-24. IA32_QM_EVTSEL and IA32_QM_CTR MSRs 


17.18.8 Monitoring Programming Considerations 


Figure 17-23 illustrates how system software can program IA32_QOSEVTSEL and IA32_QM_CTR to perform 
resource monitoring. 


System Software 
RMID Event ID Counter Data 
IA32_QOSEVTSEL MSR IA32_QM_ STR MSR 
63 A 32 7 Yo 63 62 0 
Reserved] RMID | Reserved | EvtID | | Monitoring Data 
— Event ID -— Availability 
'— Resource Monitoring ID —— Error 


Figure 17-25. Software Usage of Cache Monitoring Resources 


Though the field provided in IA32_QM_CTR allows for up to 62 bits of data to be returned, often a subset of bits are 
used. With Cache Monitoring Technology for instance, the number of bits used will be proportional to the base-two 
logarithm of the total cache size divided by the Upscaling Factor from CPUID. 


In Memory Bandwidth Monitoring the initial counter size is 24 bits, and retrieving the value at 1Hz or faster is suffi- 
cient to ensure at most one rollover per sampling period. Any future changes to counter width will be enumerated 
to software. 


Vol. 3B 17-49 


DEBUG, BRANCH PROFILE, TSC, AND INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) FEATURES 


17.18.8.1 Monitoring Dynamic Configuration 


Both the IA32_QM_EVTSEL and IA32_PQR_ASSOC registers are accessible and modifiable at any time during 
execution using RDMSR/WRMSR unless otherwise noted. When writing to these MSRs a #GP(0) will be generated 
if any of the following conditions occur: 


¢ A reserved bit is modified, 
¢ An RMID exceeding the maxRMID is used. 


17.18.8.2 Monitoring Operation With Power Saving Features 


Note that some advanced power management features such as deep package C-states may shrink the L3 cache 
and cause CMT occupancy count to be reduced. MBM bandwidth counts may increase due to flushing cached data 
out of L3. 


17.18.8.3 Monitoring Operation with Other Operating Modes 


The states in IA32_PQR_ASSOC and monitoring counter are unmodified across an SMI delivery. Thus, the execu- 
tion of SMM handler code and SMM handler’s data can manifest as spurious contribution in the monitored data. 


It is possible for an SMM handler to minimize the impact on of spurious contribution in the QOS monitoring counters 
by reserving a dedicated RMID for monitoring the SMM handler. Such an SMM handler can save the previously 
configured QOS Monitoring state immediately upon entering SMM, and restoring the QOS monitoring state back to 
the prev-SMM RMID upon exit. 


17.18.8.4 Monitoring Operation with RAS Features 


In general the Reliability, Availability and Serviceability (RAS) features present in Intel Platforms are not expected 
to significantly affect shared resource monitoring counts. In cases where software RAS features cause memory 
copies or cache accesses these may be tracked and may influence the shared resource monitoring counter values. 


17.19 INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION 
FEATURES 


The Intel Resource Director Technology (Intel RDT) feature set provides a set of allocation (resource control) capa- 
bilities including Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP). The Intel Xeon 
processor E5 v4 family (and a subset of communication-focused processors in the Intel Xeon E5 v3 family) intro- 
duce capabilities to configure and make use of the Cache Allocation Technology (CAT) mechanisms on the L3 cache. 
Certain Intel Atom processors also provide support for control over the L2 cache, with capabilities as described 
below. The programming interface for Cache Allocation Technology and for the more general allocation capabilities 
are described in the rest of this chapter. The CAT and CDP capabilities, where architecturally supported, may be 
detected and enumerated in software using the CPUID instruction, as described in this chapter. 


The Intel Xeon Processor Scalable Family introduces the Memory Bandwidth Allocation (MBA) feature which 
provides indirect control over the memory bandwidth available to CPU cores, and is discussed later in this chapter. 


17.19.1 Introduction to Cache Allocation Technology (CAT) 


Cache Allocation Technology enables an Operating System (OS), Hypervisor /Virtual Machine Manager (VMM) or 
similar system service management agent to specify the amount of cache space into which an application can fill 
(as a hint to hardware - certain features such as power management may override CAT settings). Specialized user- 
level implementations with minimal OS support are also possible, though not necessarily recommended (see notes 
below for OS/Hypervisor with respect to ring 3 software and virtual guests). Depending on the processor family, L2 
or L3 cache allocation capability may be provided, and the technology is designed to scale across multiple cache 
levels and technology generations. 
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Software can determine which levels are supported in a given platform programmatically using CPUID as described 
in the following sections. 


The CAT mechanisms defined in this document provide the following key features: 


¢ Amechanism to enumerate platform Cache Allocation Technology capabilities and available resource types that 
provides CAT control capabilities. For implementations that support Cache Allocation Technology, CPUID 
provides enumeration support to query which levels of the cache hierarchy are supported and specific CAT 
capabilities, such as the max allocation bitmask size, 


¢ Amechanism for the OS or Hypervisor to configure the amount of a resource available to a particular Class of 
Service via a list of allocation bitmasks, 


¢ Mechanisms for the OS or Hypervisor to signal the Class of Service to which an application belongs, and 


¢ Hardware mechanisms to guide the LLC fill policy when an application has been designated to belong to a 
specific Class of Service. 


Note that for many usages, an OS or Hypervisor may not want to expose Cache Allocation Technology mechanisms 
to Ring3 software or virtualized guests. 


The Cache Allocation Technology feature enables more cache resources (i.e. cache space) to be made available for 
high priority applications based on guidance from the execution environment as shown in Figure 17-26. The archi- 
tecture also allows dynamic resource reassignment during runtime to further optimize the performance of the high 
priority application with minimal degradation to the low priority app. Additionally, resources can be rebalanced for 
system throughput benefit across uses cases of OSes, VMMs, containers and other scenarios by managing the 
CPUID and MSR interfaces. This section describes the hardware and software support required in the platform 
including what is required of the execution environment (i.e. OS/VMM) to support such resource control. Note that 
in Figure 17-26 the L3 Cache is shown as an example resource. 


Without CAT With CAT 


Hi Pri App Lo Pri App Hi Pri App Lo Pri App 


Shared LLC, Low priority got more cache Shared LLC, High priority got more cache 


Figure 17-26. Cache Allocation Technology Enables Allocation of More Resources to High Priority Applications 


17.19.2 Cache Allocation Technology Architecture 


The fundamental goal of Cache Allocation Technology is to enable resource allocation based on application priority 
or Class of Service (COS or CLOS). The processor exposes a set of Classes of Service into which applications (or 
individual threads) can be assigned. Cache allocation for the respective applications or threads is then restricted 
based on the class with which they are associated. Each Class of Service can be configured using capacity bitmasks 
(CBMs) which represent capacity and indicate the degree of overlap and isolation between classes. For each logical 
processor there is a register exposed (referred to here as the IA32_PQR_ASSOC MSR or PQR) to allow the OS/VMM 
to specify a COS when an application, thread or VM is scheduled. 


The usage of Classes of Service (COS) are consistent across resources and a COS may have multiple resource 
control attributes attached, which reduces software overhead at context swap time. Rather than adding new types 
of COS tags per resource for instance, the COS management overhead is constant. Cache allocation for the indi- 
cated application/thread/container/VM is then controlled automatically by the hardware based on the class and the 
bitmask associated with that class. Bitmasks are configured via the IA32_resourceType_MASK_n MSRs, where 
resourceType indicates a resource type (e.g. “L3” for the L3 cache) and “n” indicates a COS number. 


The basic ingredients of Cache Allocation Technology are as follows: 
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¢ An architecturally exposed mechanism using CPUID to indicate whether CAT is supported, and what resource 
types are available which can be controlled, 


° For each available resourceType, CPUID also enumerates the total number of Classes of Services and the length 
of the capacity bitmasks that can be used to enforce cache allocation to applications on the platform, 


¢ Anarchitecturally exposed mechanism to allow the execution environment (OS/VMM) to configure the behavior 
of different classes of service using the bitmasks available, 


¢ An architecturally exposed mechanism to allow the execution environment (OS/VMM) to assign a COS to an 
executing software thread (i.e. associating the active CR3 of a logical processor with the COS in 
IA32_PQR_ASSOC), 


¢ Implementation-dependent mechanisms to indicate which COS is associated with a memory access and to 
enforce the cache allocation on a per COS basis. 


A capacity bitmask (CBM) provides a hint to the hardware indicating the cache space an application should be 
limited to as well as providing an indication of overlap and isolation in the CAT-capable cache from other applica- 
tions contending for the cache. The bit length of the capacity mask available generally depends on the configuration 
of the cache and is specified in the enumeration process for CAT in CPUID (this may vary between models in a 
processor family as well). Similarly, other parameters such as the number of supported COS may vary for each 
resource type, and these details can be enumerated via CPUID. 


M7 M6 M5 M4 M3 M2 M1 MO 
coso A A A A A A A A Default Bitmask 
cos1| A A A A A A A A 
cos2| A A A A A A A A 
cos3| A A A A A A A A 
M7 M6 M5 M4 M3 M2 M1 MO 
coso A A A A A Overlapped Bitmask 
cos1 i 
cos2 B B 
cos3 A 
M7 M6 M5 M4 M3 M2 M1 MO 
coso A A A A Isolated Bitmask 
cos1 A A 
cos2 A 
cos3 A 


Figure 17-27. Examples of Cache Capacity Bitmasks 


Sample cache capacity bitmasks for a bit length of 8 are shown in Figure 17-27. Please note that all (and only) 
contiguous '1' combinations are allowed (e.g. FFFFH, OFFOH, 003CH, etc.). Attempts to program a value without 
contiguous '1's (including zero) will result in a general protection fault (#GP(0)). It is generally expected that in 
way-based implementations, one capacity mask bit corresponds to some number of ways in cache, but the specific 
mapping is implementation-dependent. In all cases, a mask bit set to '1' specifies that a particular Class of Service 
can allocate into the cache subset represented by that bit. A value of '0' in a mask bit specifies that a Class of 
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Service cannot allocate into the given cache subset. In general, allocating more cache to a given application is 
usually beneficial to its performance. 


Figure 17-27 also shows three examples of sets of Cache Capacity Bitmasks. For simplicity these are represented 
as 8-bit vectors, though this may vary depending on the implementation and how the mask is mapped to the avail- 
able cache capacity. The first example shows the default case where all 4 Classes of Service (the total number of 
COS are implementation-dependent) have full access to the cache. The second case shows an overlapped case, 
which would allow some lower-priority threads share cache space with the highest priority threads. The third case 
shows various non-overlapped partitioning schemes. As a matter of software policy for extensibility COSO should 
typically be considered and configured as the highest priority COS, followed by COS1, and so on, though there is 
no hardware restriction enforcing this mapping. When the system boots all threads are initialized to COSO, which 
has full access to the cache by default. 


Though the representation of the CBMs looks similar to a way-based mapping they are independent of any specific 
enforcement implementation (e.g. way partitioning.) Rather, this is a convenient manner to represent capacity, 
overlap and isolation of cache space. For example, executing a POPCNT instruction (population count of set bits) on 
the capacity bitmask can provide the fraction of cache space that a class of service can allocate into. In addition to 
the fraction, the exact location of the bits also shows whether the class of service overlaps with other classes of 
service or is entirely isolated in terms of cache space used. 


Enum/Confg Association Enforcement 
Enumerate OS Context Application 
Enforcement Switch Memory Request 


Configure CBM for Set Class of Service Tag with Cache 
each Class of Service in IA32_PQR Class of Service 


Gos =D 
[cosas ——; Transaction 
Cache Subsystem 


Cache Allocation 


Capacity bitmask 3 
Cos 1 Capacity bitmask 3 
COS 2 Capacity bitmask 3 


Enforce Mask 


Capacity bitmask 3 


Figure 17-28. Class of Service and Cache Capacity Bitmasks 


Figure 17-28 shows how the Cache Capacity Bitmasks and the per-logical-processor Class of Service are logically 
used to enable Cache Allocation Technology. All (and only) contiguous 1's in the CBM are permitted. The length of 
a CBM may vary from resource to resource or between processor generations and can be enumerated using CPUID. 
From the available mask set and based on the goals of the OS/VMM (shared or isolated cache, etc.) bitmasks are 
selected and associated with different classes of service. For the available Classes of Service the associated CBMs 
can be programmed via the global set of CAT configuration registers (in the case of L3 CAT, via the 
IA32_L3_MASK_n MSRs, where “n” is the Class of Service, starting from zero). In all architectural implementations 
supporting CPUID it is possible to change the CBMs dynamically, during program execution, unless stated other- 
wise by Intel. 


The currently running application's Class of Service is communicated to the hardware through the per-logical- 
processor PQR MSR (IA32_PQR_ASSOC MSR). When the OS schedules an application thread on a logical processor, 
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the application thread is associated with a specific COS (i.e. the corresponding COS in the PQR) and all requests to 
the CAT-capable resource from that logical processor are tagged with that COS (in other words, the application 
thread is configured to belong to a specific COS). The cache subsystem uses this tagged request information to 
enforce QoS. The capacity bitmask may be mapped into a way bitmask (or a similar enforcement entity based on 
the implementation) at the cache before it is applied to the allocation policy. For example, the capacity bitmask can 
be an 8-bit mask and the enforcement may be accomplished using a 16-way bitmask for a cache enforcement 
implementation based on way partitioning. 


The following sections describe extensions of CAT such as Code and Data Prioritization (CDP), followed by details 
on specific features such as L3 CAT, L3 CDP, L2 CAT, and L2 CDP. Depending on the specific processor a mix of 
features may be supported, and CPUID provides enumeration capabilities to enable software to dynamically detect 
the set of supported features. 


17.19.3 Code and Data Prioritization (CDP) Technology 


Code and Data Prioritization Technology is an extension of CAT. CDP enables isolation and separate prioritization of 
code and data fetches to the L2 or L3 cache in a software configurable manner, depending on hardware support, 
which can enable workload prioritization and tuning of cache capacity to the characteristics of the workload. CDP 
extends Cache Allocation Technology (CAT) by providing separate code and data masks per Class of Service (COS). 
Support for the L2 CDP feature and the L3 CDP features are separately enumerated (via CPUID) and separately 
controlled (via remapping the L2 CAT MSRs or L3 CAT MSRs respectively). Section 17.19.6.3 and Section 17.19.7 
provide details on enumerating, controlling and enabling L3 and L2 CDP respectively, while this section provides a 
general overview. 


The L3 CDP feature was first introduced on the Intel Xeon E5 v4 family of server processors, as an extension to L3 
CAT. The L2 CDP feature is first introduced on future Intel Atom family processors, as an extension to L2 CAT. 


By default, CDP is disabled on the processor. If the CAT MSRs are used without enabling CDP, the processor oper- 
ates in a traditional CAT-only mode. When CDP is enabled, 


¢ the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for data or code fetches (see 
Figure 17-29), 


¢ the range of COS for CAT is re-indexed, with the lower-half of the COS range available for CDP. 


Using the CDP feature, virtual isolation between code and data can be configured on the L2 or L3 cache if desired, 
similar to how some processor cache levels provide separate L1 data and L1 instruction caches. 


Like the CAT feature, CDP may be dynamically configured by privileged software at any point during normal system 
operation, including dynamically enabling or disabling the feature provided that certain software configuration 
requirements are met (see Section 17.19.5). 


An example of the operating mode of CDP is shown in Figure 17-29. Shown at the top are traditional CAT usage 
models where capacity masks map 1:1 with a COS number to enable control over the cache space which a given 
COS (and thus applications, threads or VMs) may occupy. Shown at the bottom are example mask configurations 
where CDP is enabled, and each COS number maps 1:2 to two masks, one for code and one for data. This enables 
code and data to be either overlapped or isolated to varying degrees either globally or on a per-COS basis, 
depending on application and system needs. 
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Example of CAT-Only Usage - 16 bit Capacity Masks 
coso| 1 1 1 1 0;0;0;0;};0;}]0;]0}]0;,0;,0) 04 0 
cos1| O 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 Traditional 
cos2;/0/;0;0;0;0;0/); 07 0 1 1 0;0;0/;] 0}; 0] 0 | CAT 
cos3; 0;0;0);0;0;0; 07 0 1 1 1 1 1 1 1 1 
Example of Code/Data Prioritization Usage - 16 bit Capacity Masks 
COSO.Data 1 1 1 1 0;0;0;0;0;0;],0;],0}]0;,0},0 1} 0 
coso.code | 9 | 9 | 9 | 9 | ! 0};90};90},0 ;,90 ,90 , 9} 9 | 9 | O | caTwith 
cosi.Data | 9 | 9/9} 90] 0) 0 1/71 {/0]0]0 10] 0 |coP 
cosi.code | 9/09} 9/0)/0)/0;0/;/0)0)1/1)1)0;0/0)] 0 
OthercOS.Data} 9 | 09}; 0} 0} 0; 0}0) 01) 0 0 | 0 1 1 1 1 
Other COS.Code} 0 | 0} 0} 0;}0/;]0/;]0/;]0/; 0 0 | 0 1 1 1 1 


Figure 17-29. Code and Data Capacity Bitmasks of CDP 


When CDP is enabled, the existing mask space for CAT-only operation is split. As an example if the system supports 
16 CAT-only COS, when CDP is enabled the same MSR interfaces are used, however half of the masks correspond 
to code, half correspond to data, and the effective numberof COS is reduced by half. Code/Data masks are defined 
per-COS and interleaved in the MSR space as described in subsequent sections. 


In cases where CPUID exposes a non-even number of supported Classes of Service for the CAT or CDP features, 
software using CDP should use the lower matched pairs of code/data masks, and any upper unpaired masks should 
not be used. As an example, if CPUID exposes 5 CLOS, when CDP is enabled then two code/data pairs are available 
(masks 0/1 for CLOS[0] data/code and masks 2/3 for CLOS[1] data/code), however the upper un-paired mask 
should not be used (mask 4 in this case) or undefined behavior may result. 


17.19.4 Enabling Cache Allocation Technology Usage Flow 


Figure 17-30 illustrates the key steps for OS/VMM to detect support of Cache Allocation Technology and enable 
priority-based resource allocation for a CAT-capable resource. 
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On OS/VMM Initialization Cache Allocation Configuration On Context Switch 
CQE Capability Configure CBM Set COS for scheduled 
Enumeration per COS thread context 

CPUID[ WRMSR WRMSR 
CPUID.(7,0):EBX.15 1A32_L3_QOS_MASK_0 
CPUID.(10H,0):EBX[31:1] _ 1A32_PQR_ASSOC 
CPUID.(10H, 1):EAX[4:0] 
CPUID.(10H, 1):EDX[15:0] 1A32_L3_QOS_MASK_n 
CPUID.(10H, 1):EBX[ 


Figure 17-30. Cache Allocation Technology Usage Flow 


Enumeration and configuration of L2 CAT is similar to L3 CAT, however CPUID details and MSR addresses differ. 
Common CLOS are used across the features. 


17.19.4.1 Enumeration and Detection Support of Cache Allocation Technology 


Software can query processor support of CAT capabilities by executing CPUID instruction with EAX = 07H, ECX = 
OH as input. If CPUID.(EAX=07H, ECX=0):EBX.PQE[bit 15] reports 1, the processor supports software control over 
shared processor resources. Software must use CPUID leaf 10H to enumerate additional details of available 
resource types, classes of services and capability bitmasks. The programming interfaces provided by Cache Alloca- 
tion Technology include: 


CPUID leaf function 10H (Cache Allocation Technology Enumeration leaf) and its sub-functions provide 
information on available resource types, and CAT capability for each resource type (see Section 17.19.4.2). 


IA32_L3_MASK_n: A range of MSRs is provided for each resource type, each MSR within that range specifying 
a software-configured capacity bitmask for each class of service. For L3 with Cache Allocation support, the CBM 
is specified using one of the IA32_L3_QOS_MASK_n MSR, where 'n' corresponds to a number within the 
supported range of COS, i.e. the range between 0 and CPUID.(EAX=10H, ECX=ResID):EDX[15:0], inclusive. 
See Section 17.19.4.3 for details. 


IA32_L2_MASK_n: A range of MSRs is provided for L2 Cache Allocation Technology, enabling software control 
over the amount of L2 cache available for each CLOS. Similar to L3 CAT, a CBM is specified for each CLOS using 
the set of registers, IA32_L2_QOS_MASK_n MSR, where 'n' ranges from zero to the maximum CLOS number 
reported for L2 CAT in CPUID. See Section 17.19.4.3 for details. 


The L2 mask MSRs are scoped at the same level as the L2 cache (similarly, the L3 mask MSRs are scoped at the 
same level as the L3 cache). Software may determine which logical processors share an MSR (for instance local 
to a core, or shared across multiple cores) by performing a write to one of these MSRs and noting which logical 
threads observe the change. Example flows for a similar method to determine register scope are described in 
Section 15.5.2, “System Software Recommendation for Managing CMCI and Machine Check Resources”. 
Software may also use CPUID leaf 4 to determine the maximum number of logical processor IDs that may share 
a given level of the cache. 


IA32_PQR_ASSOC.CLOS: The IA32_PQR_ASSOC MSR provides a COS field that OS/VMM can use to assign a 
logical processor to an available COS. The set of COS are common across all allocation features, meaning that 
multiple features may be supported in the same processor without additional software COS management 
overhead at context swap time. See Section 17.19.4.4 for details. 


17.19.4.2 Cache Allocation Technology: Resource Type and Capability Enumeration 


CPUID leaf function 10H (Cache Allocation Technology Enumeration leaf) provides two or more sub-functions: 


CAT Enumeration leaf sub-function 0 enumerates available resource types that support allocation control, i.e. 
by executing CPUID with EAX=10H and ECX=OH. Each supported resource type is represented by a bit field in 
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CPUID.(EAX=10H, ECX=0):EBX[31:1]. The bit position of each set bit corresponds to a Resource ID (ResID), 
for instance ResID=1 is used to indicate L3 CAT support, and ResID=2 indicates L2 CAT support. The ResID is 
also the sub-leaf index that software must use to query details of the CAT capability of that resource type (see 
Figure 17-31). 


CPUID.(EAX=10H, ECX=0) Output: (EAX: Reserved; ECX: Reserved; EDX: Reserved) 


31 4 3 2 1 0 
alia (ae lf 
EBX Reserved BN ey cog 
A 


Figure 17-31. CPUID.(EAX=10H, ECX=OH) Available Resource Type Identification 


— For ECX>0, EAX[4:0] reports the length of the capacity bitmask length (ECX=1 or 2 for L2 CAT or L3 CAT 
respectively) using minus-one notation, e.g., a value of 15 corresponds to the capacity bitmask having 
length of 16 bits. Bits 31:5 of EAX are reserved. 


Sub-functions of CPUID.EAX=10H with a non-zero ECX input matching a supported ResID enumerate the 
specific enforcement details of the corresponding ResID. The capabilities enumerated include the length of the 
capacity bitmasks and the number of Classes of Service for a given ResID. Software should query the capability 
of each available ResID that supports CAT from a sub-leaf of leaf 10H using the sub-leaf index reported by the 
corresponding non-zero bit in CPUID.(EAX=10H, ECX=0):EBX[31:1] in order to obtain additional feature 
details. 


CAT capability for L3 is enumerated by CPUID.(EAX=10H, ECX=1H), see Figure 17-32. The specific CAT 
capabilities reported by CPUID.(EAX=10H, ECX=1) are: 


CPUID.(EAX=10H, ECX=ResID=1) Output: 

31 5 4 0 
EAX Reserved CBM_LEN 

31 0 
EBX : ‘ F a 

Bitmask of Shareable Resource with Other executing entities 

31 2.1 0 

EG Reserved 
CDP 

31 1615 0 

EDX Reserved COS_MAX 


Figure 17-32. L3 Cache Allocation Technology and CDP Enumeration 


— CPUID.(EAX=10H, ECX=ResID=1):EAX[4:0] reports the length of the capacity bitmask length using 
minus-one notation, i.e. a value of 15 corresponds to the capability bitmask having length of 16 bits. Bits 
31:5 of EAX are reserved. 


— CPUID.(EAX=10H, ECX=1):EBX[31:0] reports a bit mask. Each set bit within the length of the CBM 
indicates the corresponding unit of the L3 allocation may be used by other entities in the platform (e.g. an 
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integrated graphics engine or hardware units outside the processor core and have direct access to L3). Each 
cleared bit within the length of the CBM indicates the corresponding allocation unit can be configured to 
implement a priority-based allocation scheme chosen by an OS/VMM without interference with other 
hardware agents in the system. Bits outside the length of the CBM are reserved. 


— CPUID.(EAX=10H, ECX=1):ECX.CDP[bit 2]: If 1, indicates L3 Code and Data Prioritization Technology is 
supported (see Section 17.19.5). Other bits of CPUID.(EAX=10H, ECX=1):ECX are reserved. 


— CPUID.(EAX=10H, ECX=1):EDX[15:0] reports the maximum COS supported for the resource (COS are 
zero-referenced, meaning a reported value of '15' would indicate 16 total supported COS). Bits 31:16 are 
reserved. 


* CAT capability for L2 is enumerated by CPUID.(EAX=10H, ECX=2H), see Figure 17-33. The specific CAT 
capabilities reported by CPUID.(EAX=10H, ECX=2) are: 


CPUID.(EAX=10H, ECX=ResID=2) Output: 


31 5 4 0 
EAX Reserved CBM_LEN 
31 0 
EBX . f ; sa 
Bitmask of Shareable Resource with Other executing entities 
31 21 £0 
ECX 
Reserved 
CDP 
31 1615 0 
EDX Reserved COS_MAX 


Figure 17-33. L2 Cache Allocation Technology 


— CPUID.(EAX=10H, ECX=ResID=2):EAX[4:0] reports the length of the capacity bitmask length using 
minus-one notation, i.e. a value of 15 corresponds to the capability bitmask having length of 16 bits. Bits 
31:5 of EAX are reserved. 


— CPUID.(EAX=10H, ECX=2):EBX[31:0] reports a bit mask. Each set bit within the length of the CBM 
indicates the corresponding unit of the L2 allocation may be used by other entities in the platform. Each 
cleared bit within the length of the CBM indicates the corresponding allocation unit can be configured to 
implement a priority-based allocation scheme chosen by an OS/VMM without interference with other 
hardware agents in the system. Bits outside the length of the CBM are reserved. 


— CPUID.(EAX=10H, ECX=2):ECX.CDP[bit 2]: If 1, indicates L2 Code and Data Prioritization Technology is 
supported (see Section 17.19.6). Other bits of CPUID.(EAX=10H, ECX=2):ECX are reserved. 


— CPUID.(EAX=10H, ECX=2):EDX[15:0] reports the maximum COS supported for the resource (COS are 
zero-referenced, meaning a reported value of '15' would indicate 16 total supported COS). Bits 31:16 are 
reserved. 


A note on migration of Classes of Service (COS): Software should minimize migrations of COS across logical 
processors (across threads or cores), as a reduction in the performance of the Cache Allocation Technology feature 
may result if COS are migrated frequently. This is aligned with the industry-standard practice of minimizing unnec- 
essary thread migrations across processor cores in order to avoid excessive time spent warming up processor 
caches after a migration. In general, for best performance, minimize thread migration and COS migration across 
processor logical threads and processor cores. 
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17.19.4.3 Cache Allocation Technology: Cache Mask Configuration 


After determining the length of the capacity bitmasks (CBM) and number of COS supported using CPUID (see 
Section 17.19.4.2), each COS needs to be programmed with a CBM to dictate its available cache via a write to the 
corresponding IA32_resourceType_MASK_n register, where 'n' corresponds to a number within the supported 
range of COS, i.e. the range between 0 and CPUID.(EAX=10H, ECX=ResID):EDX[15:0], inclusive, and 
‘resourcelype' corresponds to a specific resource as enumerated by the set bits of CPUID.(EAX=10H, 
ECX=0):EAX[31:1], for instance, ‘L2’ or ‘L3’ cache. 


A hierarchy of MSRs is reserved for Cache Allocation Technology registers of the form 
IA32_resourceType_MASK_n: 


¢ From OC9OH through OD8FH (inclusive), providing support for multiple sub-ranges to support varying resource 
types. The first supported resourceType is 'L3', corresponding to the L3 cache in a platform. The MSRs range 
from OC9OH through ODOFH (inclusive), enables support for up to 128 L3 CAT Classes of Service. 


63 31 10 9 0 
cos Reserved | RMID 1A32_ PQR_ASSOC 
63 32 31 0 
Reserved Bit_ Mask 1A32_L3_ MASK_0 
63 ~ 3231 0 
Reserved | Bit_Mask 1A32_L3_MASK_n 


Figure 17-34. IA32_PQR_ASSOC, IA32_L3_MASK_n MSRs 


¢ Within the same CAT range hierarchy, another set of registers is defined for resourceType 'L2', corresponding 
to the L2 cache in a platform, and MSRs IA32_L2_MASK_n are defined for n=[0,63] at addresses OD10H 
through OD4FH (inclusive). 


Figure 17-34 and Figure 17-35 provide an overview of the relevant registers. 


63 32 31 0 


Reserved Bit_Mask 1A32_L2_MASK_0 


63 ee 32 31 0 


Bit_Mask 1A32_L2_MASK_n 


Reserved | 


Figure 17-35. IA32_L2_MASK_n MSRs 


All CAT configuration registers can be accessed using the standard RDMSR / WRMSR instructions. 


Note that once L3 or L2 CAT masks are configured, threads can be grouped into Classes of Service (COS) using the 
IA32_PQR_ASSOC MSR as described in Chapter 17, “Class of Service to Cache Mask Association: Common Across 
Allocation Features”. 


17.19.4.4 Class of Service to Cache Mask Association: Common Across Allocation Features 


After configuring the available classes of service with the preferred set of capacity bitmasks, the OS/VMM can set 
the IA32_PQR_ASSOC.COS of a logical processor to the class of service with the desired CBM when a thread 
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context switch occurs. This allows the OS/VMM to indicate which class of service an executing thread/VM belongs 
within. Each logical processor contains an instance of the IA32_PQR_ASSOC register at MSR location OC8FH, and 
Figure 17-34 shows the bit field layout for this register. Bits[63:32] contain the COS field for each logical processor. 


Note that placing the RMID field within the same PQR register enables both RMID and CLOS to be swapped at 
context swap time for simultaneous use of monitoring and allocation features with a single register write for effi- 
ciency. 


When CDP is enabled, Specifying a COS value in IA32_PQR_ASSOC.COS greater than MAX_COS_CDP =( 
CPUID.(EAX=10H, ECX=1):EDX[15:0] >> 1) will cause undefined performance impact to code and data fetches. 
In all cases, code and data masks for L2 and L3 CDP should be programmed with at least one bit set. 


Note that if the IA32_PQR_ASSOC.COS is never written then the CAT capability defaults to using COS 0, which in 
turn is set to the default mask in IA32_L3_MASK_0O - which is all “1"s (on reset). This essentially disables the 
enforcement feature by default or for legacy operating systems and software. 


See Section 17.19.7, “Introduction to Memory Bandwidth Allocation” for important COS programming consider- 
ations including maximum values when using CAT and CDP. 


17.19.5 Code and Data Prioritization (CDP): Enumerating and Enabling L3 CDP Technology 


L3 CDP is an extension of L3 CAT. The presence of the L3 CDP feature is enumerated via CPUID.(EAX=10H, 
ECX=1):ECX.CDP[bit 2] (see Figure 17-32). Most of the CPUID.(EAX=10H, ECX=1) sub-leaf data that applies to 
CAT also apply to CDP. However, CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT specifies the maximum COS 
applicable to CAT-only operation. For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, 
ECX=1):EDX.COS_MAX_CAT >>1). 


If CPUID.(EAX=10H, ECX=1):ECX.CDP[bit 2] =1, the processor supports CDP and provides a new MSR 
IA32_L3_QOS_CFG at address 0C81H. The layout of IA32_L3_QOS_CFG is shown in Figure 17-36. The bit field 
definition of IA32_L3_QOS_CFG are: 


¢ Bit 0: L3 CDP Enable. If set, enables CDP, maps CAT mask MSRs into pairs of Data Mask and Code Mask MSRs. 
The maximum allowed value to write into IA32_PQR_ASSOC.COS is COS_MAX_CDP. 


¢ Bits 63:1: Reserved. Attempts to write to reserved bits result in a #GP(0O). 


IA32_L3_QOS_CFG 
63 321 0 


Reserved 


L_ L3 CDP Enable 


Figure 17-36. Layout of IA32_L3_QOS_CFG 


IA32_L3_QOS_CFG default values are all Os at RESET, the mask MSRs are all 1s. Hence, all logical processors are 
initialized in COSO allocated with the entire L3 with CDP disabled, until software programs CAT and CDP. The scope 
of the IA32_L3_QOS_CFG MSR is defined to be the same scope as the L3 cache (e.g., typically per processor 
socket). Refer to Section 17.19.7 for software considerations while enabling or disabling L3 CDP. 


17.19.5.1 Mapping Between L3 CDP Masks and CAT Masks 


When CDP is enabled, the existing CAT mask MSR space is re-mapped to provide a code mask and a data mask per 
COS. The re-mapping is shown in Table 17-19. 
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Table 17-19. Re-indexing of COS Numbers and Mapping to CAT/CDP Mask MSRs 


Mask MSR CAT-only Operation CDP Operation 
IA32_L3_QOS_Mask_0 COSsO COSO.Data 
IA32_L3_QOS_Mask_1 cosi COS0.Code 
IA32_L3_QOS_Mask_2 COS2 COS1.Data 
IA32_L3_QOS_Mask_3 COS3 COS1.Code 
IA32_L3_QOS_Mask_4 COS4 COS2.Data 
IA32_L3_QOS_Mask_5 Coss COS2.Code 
IA32_L3_QOS_Mask_’2n’ COS’2n’ COS'n’.Data 
IA32_L3_QOS_Mask_’2n+1’ COS’2n+1’ COS’n’.Code 


One can derive the MSR address for the data mask or code mask for a given COS number ‘n’ by: 
¢ data_mask_address (n) = base + (n <<1), where base is the address of IA32_L3_QOS_MASK_0O. 
® code_mask_address (n) = base + (n <<1) +1. 


When CDP is enabled, each COS is mapped 1:2 with mask MSRs, with one mask enabling programmatic control 
over data fill location and one mask enabling control over code placement. A variety of overlapped and isolated 
mask configurations are possible (see the example in Figure 17-29). 


Mask MSR field definitions remain the same. Capacity masks must be formed of contiguous set bits, with a length 
of 1 bit or longer and should not exceed the maximum mask length specified in CPUID. As examples, valid masks 
on a cache with max bitmask length of 16b (from CPUID) include OxFFFF, OxFFOO, OxOOFF, OxOOFO, 0x0001, 
0x0003 and so on. Maximum valid mask lengths are unchanged whether CDP is enabled or disabled, and writes of 
invalid mask values may lead to undefined behavior. Writes to reserved bits will generate #GP(0). 


17.19.6 Code and Data Prioritization (CDP): Enumerating and Enabling L2 CDP Technology 


L2 CDP is an extension of the L2 CAT feature. The presence of the L2 CDP feature is enumerated via 
CPUID.(EAX=10H, ECX=2):ECX.CDP[bit 2] (see Figure 17-33). Most of the CPUID.(EAX=10H, ECX=2) sub-leaf 
data that applies to CAT also apply to CDP. However, CPUID.(EAX=10H, ECX=2):EDX.COS_MAX_CAT specifies the 
maximum COS applicable to CAT-only operation. For CDP operations, COS_MAX_CDP is equal to 
(CPUID.(EAX=10H, ECX=2):EDX.COS_MAX_CAT >>1). 


If CPUID.(EAX=10H, ECX=2):ECX.CDP[bit 2] =1, the processor supports L2 CDP and provides a new MSR 
IA32_L2_QOS_CFG at address 0C82H. The layout of IA32_L2_QOS_CFG is shown in Figure 17-37. The bit field 
definition of IA32_L2_QOS_CFG are: 


¢ Bit 0: L2 CDP Enable. If set, enables CDP, maps CAT mask MSRs into pairs of Data Mask and Code Mask MSRs. 
The maximum allowed value to write into IA32_PQR_ASSOC.COS is COS_MAX_CDP. 


° Bits 63:1: Reserved. Attempts to write to reserved bits result ina #GP(0). 


IA32_L2_QOS_CFG 
63 3.21.0 


Reserved 


L_ L2 CDP Enable 


Figure 17-37. Layout of IA32_L2_QOS_CFG 


Vol. 3B 17-61 


DEBUG, BRANCH PROFILE, TSC, AND INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) FEATURES 


IA32_L2_QOS_CFG default values are all Os at RESET, and the mask MSRs are all 1s. Hence all logical processors 
are initialized in COSO allocated with the entire L2 available and with CDP disabled, until software programs CAT 
and CDP. The IA32_L2_QOS_CFG MSR is defined at the same scope as the L2 cache, typically at the module level 
for Intel Atom processors for instance. In processors with multiple modules present it is recommended to program 
the IA32_L2_QOS_CFG MSR consistently across all modules for simplicity. 


17.19.6.1 Mapping Between L2 CDP Masks and L2 CAT Masks 


When CDP is enabled, the existing CAT mask MSR space is re-mapped to provide a code mask and a data mask per 
COS. This remapping is the same as the remapping shown in Table 17-19 for L3 CDP, but for the L2 MSR block 
(IA32_L2_QOS_MASK_n) instead of the L3 MSR block (IA32_L3_QOS_MASK_n). The same code / data mask 
mapping algorithm applies to remapping the MSR block between code and data masks. 


As with L3 CDP, when L2 CDP is enabled, each COS is mapped 1:2 with mask MSRs, with one mask enabling 
programmatic control over data fill location and one mask enabling control over code placement. A variety of over- 
lapped and isolated mask configurations are possible (see the example in Figure 17-29). 


Mask MSR field definitions for L2 CDP remain the same as for L2 CAT. Capacity masks must be formed of contiguous 
set bits, with a length of 1 bit or longer and should not exceed the maximum mask length specified in CPUID. As 
examples, valid masks on a cache with max bitmask length of 16b (from CPUID) include OxFFFF, OxFFOO, OxOOFF, 
Ox0O0FO, 0x0001, 0x0003 and so on. Maximum valid mask lengths are unchanged whether CDP is enabled or 
disabled, and writes of invalid mask values may lead to undefined behavior. Writes to reserved bits will generate 
#GP(0). 


17.19.6.2 Common L2 and L3 CDP Programming Considerations 


Before enabling or disabling L2 or L3 CDP, software should write all 1's to all of the corresponding CAT/CDP masks 
to ensure proper behavior (e.g., the IA32_L3_QOS_Mask_n set of MSRs for the L3 CAT feature). When enabling 
CDP, software should also ensure that only COS number which are valid in CDP operation is used, otherwise unde- 
fined behavior may result. For instance in a case with 16 CAT COS, since COS are reduced by half when CDP is 
enabled, software should ensure that only COS 0-7 are in use before enabling CDP (along with writing 1's to all 
mask bits before enabling or disabling CDP). 


Software should also account for the fact that mask interpretations change when CDP is enabled or disabled, 
meaning for instance that a CAT mask for a given COS may become a code mask for a different Class of Service 
when CDP is enabled. In order to simplify this behavior and prevent unintended remapping software should 
consider resetting all threads to COS[0] before enabling or disabling CDP. 


17.19.6.3 Cache Allocation Technology Dynamic Configuration 


All Resource Director Technology (RDT) interfaces including the IA32_PQR_ASSOC MSR, CAT/CDP masks, MBA 
delay values and CQM/MBM registers are accessible and modifiable at any time during execution using 
RDMSR/WRMSR unless otherwise noted. When writing to these MSRs a #GP(0) will be generated if any of the 
following conditions occur: 


e Areserved bit is modified, 


e Accessing a QOS mask register outside the supported COS (the max COS number is specified in 
CPUID.(EAX=10H, ECX=ResID):EDX[15:0]), or 


¢ Writing a COS greater than the supported maximum (specified as the maximum value of CPUID.(EAX=10H, 
ECX=ResID):EDX[15:0] for all valid ResID values) is written to the IA32_PQR_ASSOC.CLOS field. 


When CDP is enabled, specifying a COS value in IA32_PQR_ASSOC.COS outside of the lower half of the COS space 
will cause undefined performance impact to code and data fetches due to MSR space re-indexing into code/data 
masks when CDP is enabled. 


When reading the IA32_PQR_ASSOC register the currently programmed COS on the core will be returned. 
When reading an IA32_resourceType_MASK_n register the current capacity bit mask for COS 'n' will be returned. 


As noted previously, software should minimize migrations of COS across logical processors (across threads or 
cores), aS a reduction in the accuracy of the Cache Allocation feature may result if COS are migrated frequently. 
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This is aligned with the industry standard practice of minimizing unnecessary thread migrations across processor 
cores in order to avoid excessive time spent warming up processor caches after a migration. In general, for best 
performance, minimize thread migration and COS migration across processor logical threads and processor cores. 


17.19.6.4 Cache Allocation Technology Operation With Power Saving Features 


Note that the Cache Allocation Technology feature cannot be used to enforce cache coherency, and that some 
advanced power management features such as C-states which may shrink or power off various caches within the 
system may interfere with CAT hints - in such cases the CAT bitmasks are ignored and the other features take 
precedence. If the highest possible level of CAT differentiation or determinism is required, disable any power- 
saving features which shrink the caches or power off caches. The details of the power management interfaces are 
typically implementation-specific, but can be found at Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 3C. 


If software requires differentiation between threads but not absolute determinism then in many cases it is possible 
to leave power-saving cache shrink features enabled, which can provide substantial power savings and increase 
battery life in mobile platforms. In such cases when the caches are powered off (e.g., package C-states) the entire 
cache of a portion thereof may be powered off. Upon resuming an active state any new incoming data to the cache 
will be filled subject to the cache capacity bitmasks. Any data in the cache prior to the cache shrink or power off 
may have been flushed to memory during the process of entering the idle state, however, and is not guaranteed to 
remain in the cache. If differentiation between threads is the goal of system software then this model allows 
substantial power savings while continuing to deliver performance differentiation. If system software needs 
optimal determinism then power saving modes which flush portions of the caches and power them off should be 
disabled. 


NOTE 


IA32_PQR_ASSOC is saved and restored across C6 entry/exit. Similarly, the mask register contents 
are saved across package C-state entry/exit and are not lost. 


17.19.6.5 Cache Allocation Technology Operation with Other Operating Modes 


The states in IA32_PQR_ASSOC and mask registers are unmodified across an SMI delivery. Thus, the execution of 
SMM handler code can interact with the Cache Allocation Technology resource and manifest some degree of non- 
determinism to the non-SMM software stack. An SMM handler may also perform certain system-level or power 
management practices that affect CAT operation. 


It is possible for an SMM handler to minimize the impact on data determinism in the cache by reserving a COS with 
a dedicated partition in the cache. Such an SMM handler can switch to the dedicated COS immediately upon 
entering SMM, and switching back to the previously running COS upon exit. 


17.19.6.6 Associating Threads with CAT/CDP Classes of Service 


Threads are associated with Classes of Service (CLOS) via the per-logical-processor IA32_PQR_ASSOC MSR. The 
same COS concept applies to both CAT and CDP (for instance, COS[5] means the same thing whether CAT or CDP 
is in use, and the COS has associated resource usage constraint attributes including cache capacity masks). The 
mapping of COS to mask MSRs does change when CDP is enabled, according to the following guidelines: 


¢ In CAT-only Mode - one set of bitmasks in one mask MSR control both code and data. 

— Each COS number map 1:1 with a capacity mask on the applicable resource (e.g., L3 cache). 
¢ When CDP is enabled, 

— Two mask sets exist for each COS number, one for code, one for data. 

— Masks for code/data are interleaved in the MSR address space (see Table 17-19). 
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17.19.7 Introduction to Memory Bandwidth Allocation 


The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate control over memory band- 
width available per-core, and was introduced on the Intel Xeon Processor Scalable Family. This feature provides a 
method to control applications which may be over-utilizing bandwidth relative to their priority in environments such 
as the data-center. 


The MBA feature uses existing constructs from the Resource Director Technology (RDT) feature set including 
Classes of Service (CLOS). A given CLOS used for L3 CAT for instance means the same thing as a CLOS used for 
MBA. Infrastructure such as the MSR used to associate a thread with a CLOS (the IA32_PQR_ASSOC_MSR) and 
some elements of the CPUID enumeration (such as CPUID leaf 10H) are shared. 


The high-level implementation of Memory Bandwidth Allocation is shown in Figure 17-38. 


Chip Multiprocessor Platform 


-— 
Core[0] Core[n] 
( > 
Private L2 


L Private L2 


Programmable 
Request Rate 
Controller 


Programmable 
Request Rate 


New MBA Feature 


Controller 


TY 


ah Memory 
High-Speed Interconnect Gcnivaller 
Shared L3 Cache — With CAT 
~¢——_——— Cache space available to 


high-priority application 


~t—_,__ Cache space available to 
low-priority application 


Figure 17-38. A High-Level Overview of the MBA Feature 


As shown in Figure 17-38, the MBA feature introduces a programmable request rate controller between the cores 
and the high-speed interconnect, enabling indirect control over memory bandwidth for cores over-utilizing band- 
width relative to their priority. For instance, high-priority cores may be run un-throttled, but lower priority cores 
generating an excessive amount of traffic may be throttled to enable more bandwidth availability for the high- 
priority cores. 


Since MBA uses a programmable rate controller between the cores and the interconnect, higher-level shared 
caches and memory controller, bandwidth to these caches may also be reduced, so care should be taken to throttle 
only bandwidth-intense applications which do not use the off-core caches effectively. 


The throttling values exposed by MBA are approximate, and are calibrated to specific traffic patterns. As work-load 
characteristics vary, the throttling values provided may affect each workload differently. In cases where precise 
control is needed, the Memory Bandwidth Monitoring (MBM) feature can be used as input to a software controller 
which makes decisions about the MBA throttling level to apply. 


Enumeration and configuration details are discussed below followed by usage model considerations. 
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17.19.7.1 Memory Bandwidth Allocation Enumeration 


Similar to other RDT features, enumeration of the presence and details of the MBA feature is provided via a sub- 
leaf of the CPUID instruction. 


Key components of the enumeration are as follows. 

¢ Support for the MBA feature on the processor, and if MBA is supported, the following details: 
— Number of supported Classes of Service (CLOS) for the processor. 
— The maximum MBA delay value supported (which also implicitly provides a definition of the granularity). 
— An indication of whether the delay values which can be programmed are linearly spaced or not. 


The presence of any of the RDT features which enable control over shared platform resources is enumerated by 
executing CPUID instruction with EAX = 07H, ECX = OH as input. If CPUID.(EAX=07H, ECX=0):EBX.PQE[bit 15] 
reports 1, the processor supports software control over shared processor resources. Software may then use CPUID 
leaf 10H to enumerate additional details on the specific controls provided. 


Through CPUID leaf 10H software may determine whether MBA is supported on the platform. Specifically, as shown 
in Figure 17-31, bit 3 of the EBX register indicates whether MBA is supported on the processor, and the bit position 
(3) constitutes a Resource ID (ResID) which allows enumeration of MBA details. For instance, if bit 3 is supported 
this implies the presence of CPUID.10H.[ResID=3] as shown in Figure 17-38 which provides the following details. 


¢ CPUID.(EAX=10H, ECX=ResID=3):EAX[11:0] reports the maximum MBA throttling value supported, minus 
one. For instance, a value of 89 indicates that a maximum throttling value of 90 is supported. Additionally, in 
cases where a linear interface (see below) is supported then one hundred minus the maximum throttling value 
indicates the granularity, 10% in this example. 


¢ CPUID.(EAX=10H, ECX=ResID=3):EBX is reserved. 


¢ CPUID.(EAX=10H, ECX=ResID=3):ECX[2] reports whether the response of the delay values is linear (see 
text). 


¢ CPUID.(EAX=10H, ECX=ResID=3):EDX[15:0] reports the number of Classes of Service (CLOS) supported for 
the feature (minus one). For instance, a reported value of 15 implies a maximum of 16 supported MBA CLOS. 


The number of CLOS supported for the MBA feature may or may not align with other resources such as L3 CAT. In 
cases where the RDT features support different numbers of CLOS the lowest numerical CLOS support the common 
set of features, while higher CLOS may support a subset. For instance, if L3 CAT supports 8 CLOS while MBA 
supports 4 CLOS, all 8 CLOS would have L3 CAT masks available for cache control, but the upper 4 CLOS would not 
offer MBA support. In this case the upper 4 CLOS would not be subject to any throttling control. Software can 
manage supported resources / CLOS in order to either have consistent capabilities across CLOS by using the 
common subset or enable more flexibility by selectively applying resource control where needed based on careful 
CLOS and thread mapping. In all cases, CLOS[0] supports all RDT resource control features present on the plat- 
form. 


Discussion on the interpretation and usage of the MBA delay values is provided in Section 17.19.7.2 on MBA config- 
uration. 
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CPUID.(EAX = 10H, ECX = ResID = 3) Output: 


31 11 0 
EAX Reserved MBA_MAX-1 

31 0 
EBX Reserved 

31 2 1 0 
ECX Reserved 


[ MBA_Lin_Rsp 


31 16 15 0 


EDX Reserved COS_MAX 


Figure 17-39. CPUID.(EAX=10H, ECX=3H) MBA Feature Details Identification 


17.19.7.2 Memory Bandwidth Allocation Configuration 


The configuration of MBA takes consists of two processes once enumeration is complete. 


¢ Association of threads to Classes of Service (CLOS) - accomplished in a common fashion across RDT features 
as described in Section 17.19.7.1 via the IA32_PQR_ASSOC MSR. As with features such as L3 CAT, software 
may update the CLOS field of the PQR MSR at context swap time in order to maintain the proper association of 
software threads to Classes of Service on the hardware. While logical processors may each be associated with 
independent CLOS, see Section 17.19.7.3 for important usage model considerations (initial versions of the MBA 
feature select the maximum delay value across threads). 


* Configuration of the per-CLOS delay values, accomplished via the IA32_L2_QoS_Ext_BW_Thrtl_n MSR set 
shown in Table 17-20. 


The MBA delay values which may be programmed range from zero (implying zero delay, and full bandwidth avail- 
able) to the maximum (MBA_MAX) specified in CPUID as discussed in Section 17.19.7.1. The throttling values are 
approximate and do not sum to 100% across CLOS, rather they should be viewed as a maximum bandwidth “cap” 
per-CLOS. 


Software may select an MBA delay value then write the value into one or more of the 
IA32_L2_QoS_Ext_BW_Thrtl_n MSRs to update the delay values applied for a specific CLOS. As shown in Table 
17-20 the base address of the MSRs is at D50H, and the range corresponds to the maximum supported CLOS from 
CPUID.(EAX=10H, ECX=ResID=1):EDX[15:0] as described in Section 17.19.7.1. For instance, if 16 CLOS are 
supported then the valid MSR range will extend from D50H through D5F inclusive. 
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Table 17-20. MBA Delay Value MSRs 


Delay Value MSR Address 
IA32_L2_QoS_Ext_BW_Thrtl_0 D50H 

IA32_L2_QoS_Ext_BW_Thrtl_1 D51H 

IA32_L2_QoS_Ext_BW_Thrtl_2 D52H 
IA32_L2_QoS_Ext_BW_Thrtl_'COS_MAX' D50H + COS_MAX from CPUID.10H.3 


The definition for the MBA delay value MSRs is provided in Figure 17.39. The lower 16 bits are used for MBA delay 
values, and values from zero to the maximum from the CPUID MBA_MAX-1 value are supported. Values outside 
this range will generate #GP(0). 


If linear input throttling values are indicated by CPUID.(EAX=10H, ECX=ResID=3):ECX[bit 2] then values from 
zero through the MBA_MAX field from CPUID.(EAX=10H, ECX=ResID=3):EAX[11:0] are supported as inputs. In 
the linear mode the input precision is defined as 100-(MBA_MAX). For instance, if the MBA_MAX value is 90, the 
input precision is 10%. Values not an even multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10% 
delay applied). 


¢ If linear values are not supported (CPUID.(EAX=10H, ECX=ResID=3):ECX[bit 2] = 0) then input delay values 
are powers-of-two from zero to the MBA_MAX value from CPUID. In this case any values not a power of two will 
be rounded down the next nearest power of two. 


Base MSR Address = 0xD50 
63 16 15 0 
IA32_L2_QOS_Ext_BW_Thrtl_n MSR 
Reserved MBA Delay Value 


Figure 17-40. IA32_L2_QoS_Ext_BW_Thrtl_n MSR Definition 


Note that the throttling values provided to software are calibrated through specific traffic patterns, however as 
workload characteristics may vary the response precision and linearity of the delay values will vary across 
products, and should be treated as approximate values only. 


17.19.7.3. Memory Bandwidth Allocation Usage Considerations 


As the memory bandwidth control that MBA provides is indirect and approximate, using the feature with a closed- 
loop controller to also monitor memory bandwidth and how effectively the applications use the cache (via the 
Cache Monitoring Technology feature) may provide additional value. This approach also allows administrators to 
provide a band-width target or set-point which a controller could use to guide MBA throttling values applied, and 
this allows bandwidth control independent of the execution characteristics of the application. 


As control is provided per processor core (the max of the delay values of the per-thread CLOS applied to the core) 
care should be taking in scheduling threads so as to not inadvertently place a high-priority thread (with zero 
intended MBA throttling) next to a low-priority thread (with MBA throttling intended), which would lead to inadver- 
tent throttling of the high-priority thread. 
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CHAPTER 18 
PERFORMANCE MONITORING 


Intel 64 and IA-32 architectures provide facilities for monitoring performance via a PMU (Performance Monitoring 
Unit). 


NOTE 


In previous versions of this document, Chapter 19 contained “Performance Monitoring Events”. 
Chapter 19 has been removed from this document, and the performance monitoring events can 
now be found here: https://perfmon-events.intel.com/. 


Additionally, performance monitoring event files for Intel processors are hosted by the Intel Open 
Source Technology Center. These files can be downloaded here: 
https://download.01.org/perfmon/. 


18.1 PERFORMANCE MONITORING OVERVIEW 


Performance monitoring was introduced in the Pentium processor with a set of model-specific performance-moni- 
toring counter MSRs. These counters permit selection of processor performance parameters to be monitored and 
measured. The information obtained from these counters can be used for tuning system and compiler perfor- 
mance. 


In Intel P6 family of processors, the performance monitoring mechanism was enhanced to permit a wider selection 
of events to be monitored and to allow greater control events to be monitored. Next, Intel processors based on 
Intel NetBurst microarchitecture introduced a distributed style of performance monitoring mechanism and perfor- 
mance events. 


The performance monitoring mechanisms and performance events defined for the Pentium, P6 family, and Intel 
processors based on Intel NetBurst microarchitecture are not architectural. They are all model specific (not 
compatible among processor families). Intel Core Solo and Intel Core Duo processors support a set of architectural 
performance events and a set of non-architectural performance events. Newer Intel processor generations support 
enhanced architectural performance events and non-architectural performance events. 


Starting with Intel Core Solo and Intel Core Duo processors, there are two classes of performance monitoring capa- 
bilities. The first class supports events for monitoring performance using counting or interrupt-based event 
sampling usage. These events are non-architectural and vary from one processor model to another. They are 
similar to those available in Pentium M processors. These non-architectural performance monitoring events are 
specific to the microarchitecture and may change with enhancements. They are discussed in Section 18.6.3, 
“Performance Monitoring (Processors Based on Intel NetBurst® Microarchitecture).” Non-architectural events for a 
given microarchitecture cannot be enumerated using CPUID; and they can be found at: https://perfmon- 
events.intel.com/. 


The second class of performance monitoring capabilities is referred to as architectural performance monitoring. 
This class supports the same counting and Interrupt-based event sampling usages, with a smaller set of available 
events. The visible behavior of architectural performance events is consistent across processor implementations. 
Availability of architectural performance monitoring capabilities is enumerated using the CPUID.OAH. These events 
are discussed in Section 18.2. 


See also: 
— Section 18.2, “Architectural Performance Monitoring” 
— Section 18.3, “Performance Monitoring (Intel® Core™ Processors and Intel® Xeon® Processors)” 


* Section 18.3.1, “Performance Monitoring for Processors Based on Intel® Microarchitecture Code Name 
Nehalem” 


* Section 18.3.2, “Performance Monitoring for Processors Based on Intel® Microarchitecture Code Name 
Westmere” 
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* Section 18.3.3, “Intel® Xeon® Processor E7 Family Performance Monitoring Facility” 


* Section 18.3.4, “Performance Monitoring for Processors Based on Intel® Microarchitecture Code Name 
Sandy Bridge” 


* Section 18.3.5, “3rd Generation Intel® Core™ Processor Performance Monitoring Facility” 
* Section 18.3.6, “4th Generation Intel® Core™ Processor Performance Monitoring Facility” 


* Section 18.3.7, “5th Generation Intel® Core™ Processor and Intel® Core™ M Processor Performance 
Monitoring Facility” 


* Section 18.3.8, “6th Generation, 7th Generation and 8th Generation Intel® Core™ Processor 
Performance Monitoring Facility” 


¢ Section 18.3.9, “10th Generation Intel® Core™ Processor Performance Monitoring Facility” 
— Section 18.4, “Performance monitoring (Intel® Xeon™ Phi Processors)” 
° Section 18.4.1, “Intel® Xeon Phi” Processor 7200/5200/3200 Performance Monitoring” 
— Section 18.5, “Performance Monitoring (Intel Atom® Processors)” 
* Section 18.5.1, “Performance Monitoring (45 nm and 32 nm Intel Atom® Processors)” 
¢ Section 18.5.2, “Performance Monitoring for Silvermont Microarchitecture” 
¢ Section 18.5.3, “Performance Monitoring for Goldmont Microarchitecture” 
¢ Section 18.5.4, “Performance Monitoring for Goldmont Plus Microarchitecture” 
¢ Section 18.5.5, “Performance Monitoring for Tremont Microarchitecture” 
— Section 18.6, “Performance Monitoring (Legacy Intel Processors)” 
¢ Section 18.6.1, “Performance Monitoring (Intel® Core™ Solo and Intel® Core™ Duo Processors)” 
* Section 18.6.2, “Performance Monitoring (Processors Based on Intel® Core™ Microarchitecture)” 
* Section 18.6.3, “Performance Monitoring (Processors Based on Intel NetBurst® Microarchitecture)” 


¢ Section 18.6.4, “Performance Monitoring and Intel Hyper-Threading Technology in Processors Based on 
Intel NetBurst® Microarchitecture” 


¢ Section 18.6.4.5, “Counting Clocks on systems with Intel Hyper-Threading Technology in 
Processors Based on Intel NetBurst® Microarchitecture” 


¢ Section 18.6.5, “Performance Monitoring and Dual-Core Technology” 


¢ Section 18.6.6, “Performance Monitoring on 64-bit Intel Xeon Processor MP with Up to 8-MByte L3 
Cache” 


¢ Section 18.6.7, “Performance Monitoring on L3 and Caching Bus Controller Sub-Systems” 
* Section 18.6.8, “Performance Monitoring (P6 Family Processor)” 
¢ Section 18.6.9, “Performance Monitoring (Pentium Processors)” 

— Section 18.7, “Counting Clocks” 

— Section 18.8, “IA32_PERF_CAPABILITIES MSR Enumeration” 

— Section 18.9, “PEBS Facility” 


18.2 ARCHITECTURAL PERFORMANCE MONITORING 


Performance monitoring events are architectural when they behave consistently across microarchitectures. Intel 
Core Solo and Intel Core Duo processors introduced architectural performance monitoring. The feature provides a 
mechanism for software to enumerate performance events and provides configuration and counting facilities for 
events. 


Architectural performance monitoring does allow for enhancement across processor implementations. The 
CPUID.OAH leaf provides version ID for each enhancement. Intel Core Solo and Intel Core Duo processors support 
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base level functionality identified by version ID of 1. Processors based on Intel Core microarchitecture support, at 
a minimum, the base level functionality of architectural performance monitoring. Intel Core 2 Duo processor T 
7700 and newer processors based on Intel Core microarchitecture support both the base level functionality and 
enhanced architectural performance monitoring identified by version ID of 2. 


45 nm and 32 nm Intel Atom processors and Intel Atom processors based on the Silvermont microarchitecture 
support the functionality provided by versionID 1, 2, and 3; CPUID.OAH:EAX[7:0] reports versionID = 3 to indicate 
the aggregate of architectural performance monitoring capabilities. Intel Atom processors based on the Airmont 
microarchitecture support the same performance monitoring capabilities as those based on the Silvermont micro- 
architecture. 


Intel Core processors and related Intel Xeon processor families based on the Nehalem through Broadwell microar- 
chitectures support version ID 1, 2, and 3. Intel processors based on the Skylake, Kaby Lake and Coffee Lake 
microarchitectures support versionID 4. 


Next generation Intel Atom processors are based on the Goldmont microarchitecture. Intel processors based on 
the Goldmont microarchitecture support versionID 4. 


18.2.1 Architectural Performance Monitoring Version 1 


Configuring an architectural performance monitoring event involves programming performance event select regis- 
ters. There are a finite number of performance event select MSRs (IA32_PERFEVTSELx MSRs). The result of a 
performance monitoring event is reported in a performance monitoring counter (IA32_PMCx MSR). Performance 
monitoring counters are paired with performance monitoring select registers. 


Performance monitoring select registers and counters are architectural in the following respects: 

° Bit field layout of IA32_PERFEVTSELx is consistent across microarchitectures. 

e Addresses of IA32_PERFEVTSELx MSRs remain the same across microarchitectures. 

e Addresses of IA32_PMC MSRs remain the same across microarchitectures. 

¢* Each logical processor has its own set of IA32_PERFEVTSELx and IA32_PMCx MSRs. Configuration facilities and 
counters are not shared between logical processors sharing a processor core. 

Architectural performance monitoring provides a CPUID mechanism for enumerating the following information: 

¢ Number of performance monitoring counters available to software in a logical processor (each 
IA32_PERFEVTSELx MSR is paired to the corresponding IA32_PMCx MSR). 

¢ Number of bits supported in each IA32_PMCx. 

¢ Number of architectural performance monitoring events supported in a logical processor. 

Software can use CPUID to discover architectural performance monitoring availability (CPUID.OAH). The architec- 


tural performance monitoring leaf provides an identifier corresponding to the version number of architectural 
performance monitoring available in the processor. 


The version identifier is retrieved by querying CPUID.OAH:EAX[bits 7:0] (see Chapter 3, “Instruction Set Refer- 
ence, A-L,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A). If the version iden- 
tifier is greater than zero, architectural performance monitoring capability is supported. Software queries the 
CPUID.OAH for the version identifier first; it then analyzes the value returned in CPUID.OAH.EAX, CPUID.OAH.EBX 
to determine the facilities available. 


In the initial implementation of architectural performance monitoring; software can determine how many 
IA32_PERFEVTSELx/ IA32_PMCx MSR pairs are supported per core, the bit-width of PMC, and the number of archi- 
tectural performance monitoring events available. 


18.2.1.1 Architectural Performance Monitoring Version 1 Facilities 


Architectural performance monitoring facilities include a set of performance monitoring counters and performance 
event select registers. These MSRs have the following properties: 


° TA32_PMCx MSRs start at address 0C1H and occupy a contiguous block of MSR address space; the number of 
MSRs per logical processor is reported using CPUID.OAH:EAX[15:8]. Note that this may vary from the number 
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of physical counters present on the hardware, because an agent running at a higher privilege level (e.g., a 
VMM) may not expose all counters. 


IA32_PERFEVTSELx MSRs start at address 186H and occupy a contiguous block of MSR address space. Each 
performance event select register is paired with a corresponding performance counter in the OC1H address 
block. Note the number of IA32_PERFEVTSELx MSRs may vary from the number of physical counters present 
on the hardware, because an agent running at a higher privilege level (e.g., a VMM) may not expose all 
counters. 


The bit width of an IA32_PMCx MSR is reported using the CPUID.OAH:EAX[23:16]. This the number of valid bits 
for read operation. On write operations, the lower-order 32 bits of the MSR may be written with any value, and 
the high-order bits are sign-extended from the value of bit 31. 


Bit field layout of IA32_PERFEVTSELx MSRs is defined architecturally. 


See Figure 18-1 for the bit field layout of IA32_PERFEVTSELx MSRs. The bit fields are: 


Event select field (bits 0 through 7) — Selects the event logic unit used to detect microarchitectural 
conditions (see Table 18-1, for a list of architectural events and their 8-bit codes). The set of values for this field 
is defined architecturally; each value corresponds to an event logic unit for use with an architectural 
performance event. The number of architectural events is queried using CPUID.OAH:EAX. A processor may 
support only a subset of pre-defined values. 


63 31 24 2322 21 2019 18171615 87 0 
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(CMASK) , N NiclEls : Unit Mask (UMASK) Event Select 


INV—Invert counter mask —— 
EN—Enable counters 
INT—APIC interrupt enable 
PC—Pin control 
E—Edge detect 
OS—Operating system mode ——————! Reserved 
USR—User Mode 


Figure 18-1. Layout of IA32_PERFEVTSELx MSRs 


Unit mask (UMASK) field (bits 8 through 15) — These bits qualify the condition that the selected event 
logic unit detects. Valid UMASK values for each event logic unit are specific to the unit. For each architectural 
performance event, its corresponding UMASK value defines a specific microarchitectural condition. 


A pre-defined microarchitectural condition associated with an architectural event may not be applicable to a 
given processor. The processor then reports only a subset of pre-defined architectural events. Pre-defined 
architectural events are listed in Table 18-1; support for pre-defined architectural events is enumerated using 
CPUID.OAH: EBX. 


USR (user mode) flag (bit 16) — Specifies that the selected microarchitectural condition is counted when 
the logical processor is operating at privilege levels 1, 2 or 3. This flag can be used with the OS flag. 


OS (operating system mode) flag (bit 17) — Specifies that the selected microarchitectural condition is 
counted when the logical processor is operating at privilege level 0. This flag can be used with the USR flag. 


E (edge detect) flag (bit 18) — Enables (when set) edge detection of the selected microarchitectural 
condition. The logical processor counts the number of deasserted to asserted transitions for any condition that 
can be expressed by the other fields. The mechanism does not permit back-to-back assertions to be distin- 
guished. 

This mechanism allows software to measure not only the fraction of time spent in a particular state, but also the 
average length of time spent in such a state (for example, the time spent waiting for an interrupt to be 
serviced). 

PC (pin control) flag (bit 19) — Beginning with Sandy Bridge microarchitecture, this bit is reserved (not 
writeable). On processors based on previous microarchitectures, the logical processor toggles the PM/ pins and 
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increments the counter when performance-monitoring events occur; when clear, the processor toggles the PM/ 
pins when the counter overflows. The toggling of a pin is defined as assertion of the pin for a single bus clock 
followed by deassertion. 


INT (APIC interrupt enable) flag (bit 20) — When set, the logical processor generates an exception 
through its local APIC on counter overflow. 


EN (Enable Counters) Flag (bit 22) — When set, performance counting is enabled in the corresponding 
performance-monitoring counter; when clear, the corresponding counter is disabled. The event logic unit for a 
UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0, before writing to IA32_PMCx. 


INV (invert) flag (bit 23) — When set, inverts the counter-mask (CMASK) comparison, so that both greater 
than or equal to and less than comparisons can be made (0: greater than or equal; 1: less than). Note if 
counter-mask is programmed to zero, INV flag is ignored. 


Counter mask (CMASK) field (bits 24 through 31) — When this field is not zero, a logical processor 
compares this mask to the events count of the detected microarchitectural condition during a single cycle. If 
the event count is greater than or equal to this mask, the counter is incremented by one. Otherwise the counter 
is not incremented. 


This mask is intended for software to characterize microarchitectural conditions that can count multiple 
occurrences per cycle (for example, two or more instructions retired per clock; or bus queue occupations). If 
the counter-mask field is 0, then the counter is incremented each cycle by the event count associated with 
multiple occurrences. 


18.2.1.2 Pre-defined Architectural Performance Events 


Table 18-1 lists architecturally defined events. 


Table 18-1. UMask and Event Select Encodings for Pre-Defined Architectural Performance Events 


Bit Position Event Name UMask Event Select 
CPUID.AH.EBX 

0) UnHalted Core Cycles OOH 3CH 
1 Instruction Retired OOH COH 
2 UnHalted Reference Cycles! 01H 3CH 
3 LLC Reference 4FH 2EH 
4 LLC Misses 41H 2EH 
5 Branch Instruction Retired OOH C4H 
6 Branch Misses Retired 00H CSH 
7 Topdown Slots 01H A4H 
NOTES: 


1. Current implementations count at core crystal clock, TSC, or bus clock frequency. 


A processor that supports architectural performance monitoring may not support all the predefined architectural 
performance events (Table 18-1). The number of architectural events is reported through CPUID.OAH:EAX[31:24], 
while non-zero bits in CPUID.OAH:EBX indicate any architectural events that are not available. 


The behavior of each architectural performance event is expected to be consistent on all processors that support 
that event. Minor variations between microarchitectures are noted below: 


UnHalted Core Cycles — Event select 3CH, Umask 00H 


This event counts core clock cycles when the clock signal on a specific core is running (not halted). The counter 
does not advance in the following conditions: 


— anACPI C-state other than CO for normal operation 
— HLT 
— STPCLK# pin asserted 
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— being throttled by TM1 
— during the frequency switching phase of a performance state transition (see Chapter 14, “Power and 
Thermal Management”) 


The performance counter for this event counts across performance state transitions using different core clock 
frequencies 


° Instructions Retired — Event select COH, Umask 00H 


This event counts the number of instructions at retirement. For instructions that consist of multiple micro-ops, 
this event counts the retirement of the last micro-op of the instruction. An instruction with a REP prefix counts 
as one instruction (not per iteration). Faults before the retirement of the last micro-op of a multi-ops instruction 
are not counted. 

This event does not increment under VM-exit conditions. Counters continue counting during hardware 
interrupts, traps, and inside interrupt handlers. 


° UnHalted Reference Cycles — Event select 3CH, Umask 01H 


This event counts reference clock cycles at a fixed frequency while the clock signal on the core is running. The 
event counts at a fixed frequency, irrespective of core frequency changes due to performance state transitions. 
Processors may implement this behavior differently. Current implementations use the core crystal clock, TSC or 
the bus clock. Because the rate may differ between implementations, software should calibrate it to a time 
source with known frequency. 


° Last Level Cache References — Event select 2EH, Umask 4FH 


This event counts requests originating from the core that reference a cache line in the last level on-die cache. 
The event count includes speculation and cache line fills due to the first-level cache hardware prefetcher, but 
may exclude cache line fills due to other hardware-prefetchers. 


Because cache hierarchy, cache sizes and other implementation-specific characteristics; value comparison to 
estimate performance differences is not recommended. 


° Last Level Cache Misses — Event select 2EH, Umask 41H 


This event counts each cache miss condition for references to the last level on-die cache. The event count may 
include speculation and cache line fills due to the first-level cache hardware prefetcher, but may exclude cache 
line fills due to other hardware-prefetchers. 


Because cache hierarchy, cache sizes and other implementation-specific characteristics; value comparison to 
estimate performance differences is not recommended. 


¢ Branch Instructions Retired — Event select C4H, Umask 00H 


This event counts branch instructions at retirement. It counts the retirement of the last micro-op of a branch 
instruction. 


¢ All Branch Mispredict Retired — Event select CSH, Umask 00H 


This event counts mispredicted branch instructions at retirement. It counts the retirement of the last micro-op 
of a branch instruction in the architectural path of execution and experienced misprediction in the branch 
prediction hardware. 


Branch prediction hardware is implementation-specific across microarchitectures; value comparison to 
estimate performance differences is not recommended. 


* Topdown Slots — Event select A4H, Umask 01H 
This event counts the total number of available slots for an unhalted logical processor. 


The event increments by machine-width of the narrowest pipeline as employed by the Top-down Microarchi- 
tecture Analysis method. The count is distributed among unhalted logical processors (hyper-threads) who 
share the same physical core, in processors that support Intel Hyper-Threading Technology. 


Software can use this event as the denominator for the top-level metrics of the Top-down Microarchitecture 
Analysis method. 


NOTE 


Programming decisions or software precisians on functionality should not be based on the event 
values or dependent on the existence of performance monitoring events. 
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18.2.2 Architectural Performance Monitoring Version 2 


The enhanced features provided by architectural performance monitoring version 2 include the following: 


Fixed-function performance counter register and associated control register — Three of the architec- 
tural performance events are counted using three fixed-function MSRs (IA32_FIXED_CTRO through 
IA32_FIXED_CTR2). Each of the fixed-function PMC can count only one architectural performance event. 


Configuring the fixed-function PMCs is done by writing to bit fields in the MSR (IA32_FIXED_CTR_CTRL) located 
at address 38DH. Unlike configuring performance events for general-purpose PMCs (IA32_PMCx) via UMASK 
field in (IA32_PERFEVTSELx), configuring, programming IA32_FIXED_CTR_CTRL for fixed-function PMCs do 
not require any UMASK. 


Simplified event programming — Most frequent operation in programming performance events are 
enabling/disabling event counting and checking the status of counter overflows. Architectural performance 
event version 2 provides three architectural MSRs: 


— IA32_PERF_GLOBAL_CTRL allows software to enable/disable event counting of all or any combination of 
fixed-function PMCs (IA32_FIXED_CTRx) or any general-purpose PMCs via a single WRMSR. 


— IA32_PERF_GLOBAL_STATUS allows software to query counter overflow conditions on any combination of 
fixed-function PMCs or general-purpose PMCs via a single RDMSR. 


— IA32_PERF_GLOBAL_OVF_CTRL allows software to clear counter overflow conditions on any combination of 
fixed-function PMCs or general-purpose PMCs via a single WRMSR. 


PMI Overhead Mitigation — Architectural performance monitoring version 2 introduces two bit field interface 
in IA32_DEBUGCTL for PMI service routine to accumulate performance monitoring data and LBR records with 
reduced perturbation from servicing the PMI. The two bit fields are: 


— IA32_DEBUGCTL.Freeze_LBR_On_PMI(bit 11). In architectural performance monitoring version 2, only the 
legacy semantic behavior is supported. See Section 17.4.7 for details of the legacy Freeze LBRs on PMI 
control. 


— IA32_DEBUGCTL.Freeze_PerfMon_On_PMI(bit 12). In architectural performance monitoring version 2, 
only the legacy semantic behavior is supported. See Section 17.4.7 for details of the legacy Freeze LBRs on 
PMI control. 


The facilities provided by architectural performance monitoring version 2 can be queried from CPUID leaf OAH by 
examining the content of register EDX: 


Bits 0 through 4 of CPUID.OAH.EDX indicates the number of fixed-function performance counters available per 
core, 


Bits 5 through 12 of CPUID.OAH.EDX indicates the bit-width of fixed-function performance counters. Bits 
beyond the width of the fixed-function counter are reserved and must be written as zeros. 


NOTE 


Early generation of processors based on Intel Core microarchitecture may report in 
CPUID.OAH:EDX of support for version 2 but indicating incorrect information of version 2 facilities. 


The IA32_FIXED_CTR_CTRL MSR include multiple sets of 4-bit field, each 4 bit field controls the operation of a 
fixed-function performance counter. Figure 18-2 shows the layout of 4-bit controls for each fixed-function PMC. 
Two sub-fields are currently defined within each control. The definitions of the bit fields are: 
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Cntr2 — Controls for |A32_FIXED_CTR2 
Cntr1 — Controls for |A32_FIXED_CTR1 
PMI — Enable PMI on overflow 
CntrO — Controls for |A32_FIXED_CTRO 
ENABLE — 0: disable; 1: OS; 2: User; 3: All ring levels 


Reserved 


Figure 18-2. Layout of IA32_FIXED_CTR_CTRL MSR 


¢ Enable field (lowest 2 bits within each 4-bit control) — When bit 0 is set, performance counting is 
enabled in the corresponding fixed-function performance counter to increment while the target condition 
associated with the architecture performance event occurred at ring 0. When bit 1 is set, performance counting 
is enabled in the corresponding fixed-function performance counter to increment while the target condition 
associated with the architecture performance event occurred at ring greater than 0. Writing 0 to both bits stops 
the performance counter. Writing a value of 11B enables the counter to increment irrespective of privilege 
levels. 


° PMI field (the fourth bit within each 4-bit control) — When set, the logical processor generates an 
exception through its local APIC on overflow condition of the respective fixed-function counter. 


IA32_PERF_GLOBAL_CTRL MSR provides single-bit controls to enable counting of each performance counter. 
Figure 18-3 shows the layout of IA32_PERF_GLOBAL_CTRL. Each enable bit in IA32_PERF_GLOBAL_CTRL is 
AND’ed with the enable bits for all privilege levels in the respective IA32_PERFEVTSELx or 
IA32_PERF_FIXED_CTR_CTRL MSRs to start/stop the counting of respective counters. Counting is enabled if the 
AND’ed results is true; counting is disabled when the result is false. 
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1A32_FIXED_CTR2 enable 
1A32_FIXED_CTR1 enable 
1A32_FIXED_CTRO enable 
1A32_PMC1 enable 
1A32_PMCO enable 


Reserved 


Figure 18-3. Layout of IA32_PERF_GLOBAL_CTRL MSR 


The behavior of the fixed function performance counters supported by architectural performance version 2 is 
expected to be consistent on all processors that support those counters, and is defined as follows. 


18-8 Vol. 3B 


PERFORMANCE MONITORING 


Table 18-2. Association of Fixed-Function Performance Counters with Architectural Performance Events 


Fixed-Function Address | Event Mask Mnemonic Description 
Performance Counter 
|A32_FIXED_CTRO 309H INST_RETIRED.ANY This event counts the number of instructions that retire 


execution. For instructions that consist of multiple uops, 
this event counts the retirement of the last uop of the 
instruction. The counter continues counting during 
hardware interrupts, traps, and in-side interrupt handlers. 


IA32_FIXED_CTR1 30AH CPU_CLK_UNHALTED.THREAD | The CPU_CLK_UNHALTED.THREAD event counts the 
CPU CLK UNHALTED.CORE number of core cycles while the logical processor is not ina 
OO halt state. 


If there is only one logical processor in a processor core, 
CPU_CLK_UNHALTED.CORE counts the unhalted cycles of 
the processor core. 


The core frequency may change from time to time due to 
transitions associated with Enhanced Intel SpeedStep 
Technology or TM2. For this reason this event may have a 
changing ratio with regards to time. 


IA32_FIXED_CTR2 30BH CPU_CLK_UNHALTED.REF_TSC | This event counts the number of reference cycles at the 
TSC rate when the core is not in a halt state and not ina TM 
stop-clock state. The core enters the halt state when it is 
running the HLT instruction or the MWAIT instruction. This 
event is not affected by core frequency changes (e.g., P 
states) but counts at the same frequency as the time stamp 
counter. This event can approximate elapsed time while the 
core was not in a halt state and not in a TM stopclock state. 


IA32_FIXED_CTR3 30CH TOPDOWN.SLOTS This event counts the number of available slots for an 
unhalted logical processor. The event increments by 
machine-width of the narrowest pipeline as employed by 
the Top-down Microarchitecture Analysis method. The 
count is distributed among unhalted logical processors 
(hyper-threads) who share the same physical core. 
Software can use this event as the denominator for the 
top-level metrics of the Top-down Microarchitecture 
Analysis method. 


IA32_PERF_GLOBAL_STATUS MSR provides single-bit status for software to query the overflow condition of each 
performance counter. IA32_PERF_GLOBAL_STATUS[bit 62] indicates overflow conditions of the DS area data 
buffer. IA32_PERF_GLOBAL_STATUS[bit 63] provides a CondChgd bit to indicate changes to the state of perfor- 
mance monitoring hardware. Figure 18-4 shows the layout of IA32_PERF_GLOBAL_STATUS. A value of 1 in bits 0, 
1, 32 through 34 indicates a counter overflow condition has occurred in the associated counter. 


When a performance counter is configured for PEBS, overflow condition in the counter generates a performance- 
monitoring interrupt signaling a PEBS event. On a PEBS event, the processor stores data records into the buffer 
area (see Section 18.15.5), clears the counter overflow status., and sets the “OvfBuffer” bit in 
IA32_PERF_GLOBAL_STATUS. 
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CondChgd 
OvfDSBuffer — 
IA32_FIXED_CTR2 Overflow. 
IA32_FIXED_CTR1 Overflow 
IA32_FIXED_CTRO Overflow: 
IA32_PMC1 Overflow 
IA32_PMCO Overflow 


Reserved 


Figure 18-4. Layout of IA32_PERF_GLOBAL_STATUS MSR 


IA32_PERF_GLOBAL_OVF_CTL MSR allows software to clear overflow indicator(s) of any general-purpose or fixed- 
function counters via a single WRMSR. Software should clear overflow indications when 


¢ Setting up new values in the event select and/or UMASK field for counting or interrupt-based event sampling. 
* Reloading counter values to continue collecting next sample. 

¢ Disabling event counting or interrupt-based event sampling. 

The layout of IA32_PERF_GLOBAL_OVF_CTL is shown in Figure 18-5. 


6362 35 34.33 3231 210 


ClrCondChgd 


ClrOvfDSBuffer 
1A32_FIXED_CTR2 ClrOverflow 
1A32_FIXED_CTR1 ClrOverflow 
IA32_FIXED_CTRO ClrOverflow 
IA32_PMC1 ClrOverflow 
1A32_PMCO ClrOverflow 


Reserved 


Figure 18-5. Layout of |A32_PERF_GLOBAL_OVF_CTRL MSR 


18.2.3. Architectural Performance Monitoring Version 3 


Processors supporting architectural performance monitoring version 3 also supports version 1 and 2, as well as 
capability enumerated by CPUID leaf OAH. Specifically, version 3 provides the following enhancement in perfor- 
mance monitoring facilities if a processor core comprising of more than one logical processor, i.e. a processor core 
supporting Intel Hyper-Threading Technology or simultaneous multi-threading capability: 


e AnyThread counting for processor core supporting two or more logical processors. The interface that supports 
AnyThread counting include: 


— Each IA32_PERFEVTSELx MSR (starting at MSR address 186H) support the bit field layout defined in Figure 
18-6. 
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INV—Invert counter mask —— 
EN—Enable counters 
ANY—Any Thread —_ 
INT—APIC interrupt enable 
PC—Pin control 
E—Edge detect 
OS—Operating system mode———! Reserved 
USR—User Mode 


Figure 18-6. Layout of I[A32_PERFEVTSELx MSRs Supporting Architectural Performance Monitoring Version 3 


Bit 21 (AnyThread) of IA32_PERFEVTSELx is supported in architectural performance monitoring version 3 for 
processor core comprising of two or more logical processors. When set to 1, it enables counting the associated 
event conditions (including matching the thread’s CPL with the OS/USR setting of IA32_PERFEVTSELx) 
occurring across all logical processors sharing a processor core. When bit 21 is 0, the counter only increments 
the associated event conditions (including matching the thread’s CPL with the OS/USR setting of 
IA32_PERFEVTSELx) occurring in the logical processor which programmed the IA32_PERFEVTSELx MSR. 


— Each fixed-function performance counter IA32_FIXED_CTRx (starting at MSR address 309H) is configured 
by a 4-bit control block in the IA32_PERF_FIXED_CTR_CTRL MSR. The control block also allow thread- 
specificity configuration using an AnyThread bit. The layout of IA32_PERF_FIXED_CTR_CTRL MSR is 
shown. 
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Cntr2 — Controls for |A32_FIXED_CTR2 
Cntr1 — Controls for |A32_FIXED_CTR1 
PMI — Enable PMI on overflow on IA32_FIXED_CTRO 
AnyThread — AnyThread for I[A32_FIXED_CTRO 
ENABLE — IA32_FIXED_CTRO. 0: disable; 1: OS; 2: User; 3: All ring levels 


Reserved 


Figure 18-7. [|A32_FIXED_CTR_CTRL MSR Supporting Architectural Performance Monitoring Version 3 


Each control block for a fixed-function performance counter provides an AnyThread (bit position 2 + 4*N, N= 
0, 1, etc.) bit. When set to 1, it enables counting the associated event conditions (including matching the 
thread’s CPL with the ENABLE setting of the corresponding control block of IA32_PERF_FIXED_CTR_CTRL) 
occurring across all logical processors sharing a processor core. When an AnyThread bit is 0 in 
IA32_PERF_FIXED_CTR_CTRL, the corresponding fixed counter only increments the associated event 
conditions occurring in the logical processor which programmed the IA32_PERF_FIXED_CTR_CTRL MSR. 


The IA32_PERF_GLOBAL_CTRL, IA32_PERF_GLOBAL_STATUS, IA32_PERF_GLOBAL_OVF_CTRL MSRs provide 
single-bit controls/status for each general-purpose and fixed-function performance counter. Figure 18-8 and 
Figure 18-9 show the layout of these MSRs for N general-purpose performance counters (where N is reported 
by CPUID.OAH:EAX[15:8]) and three fixed-function counters. 
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NOTE 


The number of general-purpose performance monitoring counters (i.e., N in Figure 18-9) can vary 
across processor generations within a processor family, across processor families, or could be 
different depending on the configuration chosen at boot time in the BIOS regarding Intel Hyper 
Threading Technology, (e.g. N=2 for 45 nm Intel Atom processors; N =4 for processors based on 
the Nehalem microarchitecture; for processors based on the Sandy Bridge microarchitecture, N = 
4 if Intel Hyper Threading Technology is active and N=8 if not active). In addition, the number of 
counters may vary from the number of physical counters present on the hardware, because an 
agent running at a higher privilege level (e.g., a VMM) may not expose all counters. 


Global Enable Controls IA32_PERF_GLOBAL_CTRL 
63 35 34 33 32 31 Ni. s 10 


Reserved 


1A32_FIXED_CTR2 enable 
1A32_FIXED_CTR1 enable 
1A32_FIXED_CTRO enable 
1A32_PMC(N-1) enable 
sgepneeeemiieee enable 
1A32_PMC1 enable 
1A32_PMCO enable 


Figure 18-8. Layout of Global Performance Monitoring Control MSR 


Global Overflow Status [A32_PERF_GLOBAL_STATUS 
636261 35 3433 3231 N.. 10 
OvfDSBuffer 


OvfUncore 

1A32_FIXED_CTR2 Overflow §=————____1 1A32_PMC(N-1) Overflow 
IA32_FIXED_CTR1 Overflow § fpr Overflo' 
IA32_FIXED_CTRO Overflow 
1A32_PMC1 Overflow 
1A32_PMCO Overflow 


CondChgd —— 


Global Overflow Status [A32_PERF_GLOBAL_OVF_CTRL 
6362 35 3433 3231 Ni... 10 


ClrCondChgd —_ 
ClrOvfDSBuffer —J 
ClrOvfUncore 
1A32_FIXED_CTR2 ClrOverflow 
1A32_FIXED_CTR1 ClrOverflow 
1A32_FIXED_CTRO ClrOverflow 
IA32_PMC1 ClrOverflow 
IA32_PMCO ClrOverflow 


1A32_PMC(N-1) ClrOverflow 
i bine eke ClrOverflow——______ 


Figure 18-9. Global Performance Monitoring Overflow Status and Control MSRs 
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18.2.3.1 AnyThread Counting and Software Evolution 


The motivation for characterizing software workload over multiple software threads running on multiple logical 
processors of the same processor core originates from a time earlier than the introduction of the AnyThread inter- 
face in IA32_PERFEVTSELx and IA32_FIXED_CTR_CTRL. While AnyThread counting provides some benefits in 
simple software environments of an earlier era, the evolution contemporary software environments introduce 
certain concepts and pre-requisites that AnyThread counting does not comply with. 

One example is the proliferation of software environments that support multiple virtual machines (VM) under VMX 
(see Chapter 22, “Introduction to Virtual-Machine Extensions”) where each VM represents a domain separated 
from one another. 

A Virtual Machine Monitor (VMM) that manages the VMs may allow an individual VM to employ performance moni- 
toring facilities to profiles the performance characteristics of a workload. The use of the Anythread interface in 
IA32_PERFEVTSELx and IA32_FIXED_CTR_CTRL is discouraged with software environments supporting virtualiza- 
tion or requiring domain separation. 


Specifically, Intel recommends VMM: 


° Configure the MSR bitmap to cause VM-exits for WRMSR to IA32_PERFEVTSELx and IA32_FIXED_CTR_CTRL in 
VMX non-Root operation (see CHAPTER 23 for additional information), 

° Clear the AnyThread bit of IA32_PERFEVTSELx and IA32_FIXED_CTR_CTRL in the MSR-load lists for VM exits 
and VM entries (see CHAPTER 23, CHAPTER 25, and CHAPTER 26). 


Even when operating in simpler legacy software environments which might not emphasize the pre-requisites of a 
virtualized software environment, the use of the AnyThread interface should be moderated and follow any event- 
specific guidance where explicitly noted. 


18.2.4 Architectural Performance Monitoring Version 4 


Processors supporting architectural performance monitoring version 4 also supports version 1, 2, and 3, as well as 

capability enumerated by CPUID leaf OAH. Version 4 introduced a streamlined PMI overhead mitigation interface 

that replaces the legacy semantic behavior but retains the same control interface in 

IA32_DEBUGCTL.Freeze_LBRs_On_PMI and Freeze_PerfMon_On_PMI. Specifically version 4 provides the following 

enhancements: 

¢ New indicators (LBR_FRZ, CTR_FRZ) in IA32_PERF_GLOBAL_STATUS, see Section 18.2.4.1. 

¢ Streamlined Freeze/PMI Overhead management interfaces to use IA32_DEBUGCTL.Freeze_LBRs_On_PMI and 
IA32_DEBUGCTL.Freeze_PerfMon_On_PMI: see Section 18.2.4.1. Legacy semantics of Freeze_LBRs_On_PMI 
and Freeze_PerfMon_On_PMI (applicable to version 2 and 3) are not supported with version 4 or higher. 


¢  Fine-grain separation of control interface to manage overflow/status of IA32_PERF_GLOBAL_STATUS and 
read-only performance counter enabling interface in IA32_PERF_GLOBAL_STATUS: see Section 18.2.4.2. 


¢ Performance monitoring resource in-use MSR to facilitate cooperative sharing protocol between perfmon- 
managing privilege agents. 


18.2.4.1 Enhancement in |IA32_PERF_GLOBAL_STATUS 


The IA32_PERF_GLOBAL_STATUS MSR provides the following indicators with architectural performance monitoring 
version 4: 
° TA32_PERF_GLOBAL_STATUS.LBR_FRZ[bit 58]: This bit is set due to the following conditions: 

— IA32_DEBUGCTL.FREEZE_LBR_ON_PMI has been set by the profiling agent, and 


— A performance counter, configured to generate PMI, has overflowed to signal a PMI. Consequently the LBR 
stack is frozen. 


Effectively, the IA32_PERF_GLOBAL_STATUS.LBR_FRZ bit also serves as a control to enable capturing data in 
the LBR stack. To enable capturing LBR records, the following expression must hold with architectural perfmon 
version 4 or higher: 


— (IA32_DEBUGCTL.LBR & (!IA32_PERF_GLOBAL_STATUS.LBR_FRZ) ) =1 
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e = JTA32_PERF_GLOBAL_STATUS.CTR_FRZ[bit 59]: This bit is set due to the following conditions: 
— JA32_DEBUGCTL.FREEZE_PERFMON_ON_PMI has been set by the profiling agent, and 


— A performance counter, configured to generate PMI, has overflowed to signal a PMI. Consequently, all the 
performance counters are frozen. 


Effectively, the IA32_PERF_GLOBAL_STATUS.CTR_FRZ bit also serve as an read-only control to enable 
programmable performance counters and fixed counters in the core PMU. To enable counting with the 
performance counters, the following expression must hold with architectural perfmon version 4 or higher: 


¢ (IJA32_PERFEVTSELn.EN & IA32_PERF_GLOBAL_CTRL.PMCn & 
(1IA32_PERF_GLOBAL_STATUS.CTR_FRZ) ) = 1 for programmable counter ‘n’, or 


¢ (IJA32_PERF_FIXED_CRTL.ENi & IA32_PERF_GLOBAL_CTRL.FCi & 
(1IA32_PERF_GLOBAL_STATUS.CTR_FRZ) ) = 1 for fixed counter ‘i’ 


The read-only enable interface IA32_PERF_GLOBAL_STATUS.CTR_FRZ provides a more efficient flow for a PMI 
handler to use IA32_ DEBUGCTL.Freeze_Perfmon_On_PMI to filter out data that may distort target workload anal- 
ysis, see Table 17-3. It should be noted the IA32_PERF_GLOBAL_CTRL register continue to serve as the primary 
interface to control all performance counters of the logical processor. 


For example, when the Freeze-On-PMI mode is not being used, a PMI handler would be setting 
IA32_PERF_GLOBAL_CTRL as the very last step to commence the overall operation after configuring the individual 
counter registers, controls and PEBS facility. This does not only assure atomic monitoring but also avoids unneces- 
sary complications (e.g. race conditions) when software attempts to change the core PMU configuration while some 
counters are kept enabled. 


Additionally, IA32_PERF_GLOBAL_STATUS.TraceToPAPMI [bit 55]: On processors that support Intel Processor Trace 
and configured to store trace output packets to physical memory using the ToPA scheme, bit 55 is set when a PMI 
occurred due to a ToPA entry memory buffer was completely filled. 


IA32_PERF_GLOBAL_STATUS also provides an indicator to distinguish interaction of performance monitoring oper- 
ations with other side-band activities, which apply Intel SGX on processors that support SGX (For additional infor- 
mation about Intel SGX, see “Intel® Software Guard Extensions Programming Reference”.): 


e = TA32_PERF_GLOBAL_STATUS.ASCI[bit 60]: This bit is set when data accumulated in any of the configured 
performance counters (i.e. IA32_PMCx or IA32_FIXED_CTRx) may include contributions from direct or indirect 
operation of Intel SGX to protect an enclave (since the last time IA32_PERF_GLOBAL_STATUS.ASCI was 
cleared). 


6362 61 6059 58 55 35 34 33 32 31 N... 10 


CondChgd 
OvfDSBuffer 
OvfUncore 
ASCI IA32_PMC(N-1) Overfl 
CTR Fre. AAA ff rete ttt ttttttteee 
LBR_Frz 1A32_PMC1 Overflow 


Trace ToPAPMII 1A32_PMCO Overflow 
IA32_ FIXED_CTR2 Overflow 7 


IA32_FIXED_CTR1 Overflow Reserved 
1A32_FIXED_CTRO Overflow 


Figure 18-10. [A32_PERF_GLOBAL_STATUS MSR and Architectural Perfmon Version 4 


Note, a processor’s support for IA32_PERF_GLOBAL_STATUS. TraceToPAPMI[bit 55] is enumerated as a result of 
CPUID enumerated capability of Intel Processor Trace and the use of the ToPA buffer scheme. Support of 
IA32_PERF_GLOBAL_STATUS.ASCI[bit 60] is enumerated by the CPUID enumeration of Intel SGX. 
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18.2.4.2 1A32_PERF_GLOBAL_STATUS_RESET and IA32_PERF_GLOBAL_STATUS_SET MSRS 


With architectural performance monitoring version 3 and lower, clearing of the set bits in 
IA32_PERF_GLOBAL_STATUS MSR by software is done via IA32_PERF_GLOBAL_OVF_CTRL MSR. Starting with 
architectural performance monitoring version 4, software can manage the overflow and other indicators in 
IA32_PERF_GLOBAL_STATUS using separate interfaces to set or clear individual bits. 

The address and the architecturally-defined bits of IA32_PERF_GLOBAL_OVF_CTRL is inherited by 


IA32_PERF_GLOBAL_STATUS_RESET (see Figure 18-11). Further, IA32_PERF_GLOBAL_STATUS_RESET provides 
additional bit fields to clear the new indicators in IA32_PERF_GLOBAL_STATUS described in Section 18.2.4.1. 


6362 61 6059 58 55 35 3433 32 31 NN... 10 


Clr CondChgd 
Clr OvfDSBuffer 
Clr OvfUncore 
Clr ASCI Clr 1A32_PMC(N-1) Ovf 
Clr CTR_Frz GIP esyetinteieecedsczdzee, Ovf 
Clr LBR_Frz Clr |A32_PMC1 Ovf 
Clr TraceToPAPMI Clr |A32_PMCO Ovt 

Clr |A32_FIXED_CTR2 Ovf 
Clr |A32_FIXED_CTR1 Ovf Reserved 
Clr |A32_FIXED_CTRO Ovf 


Figure 18-11. [A32_PERF_GLOBAL_STATUS_RESET MSR and Architectural Perfmon Version 4 


The IA32_PERF_GLOBAL_STATUS_SET MSR is introduced with architectural performance monitoring version 4. It 
allows software to set individual bits in IA32_PERF_GLOBAL_STATUS. The IA32_PERF_GLOBAL_STATUS_SET 
interface can be used by a VMM to virtualize the state of IA32_PERF_GLOBAL_STATUS across VMs. 


63 62 61 60 59 58 55 35 34 33 32 31 NN... 10 
Set OvfDSBuffer 


Set OvfUncore ———_ 
Set ASCI Set IA32_PMC(N-1) Ovf 


Set CTR_Frz Sethekk eeae Ovf 
Set LBR_Frz Set IA32_PMC1 Ovf 
Set TraceToPAPMI Set IA32_PMCO Ovf 
Set IA32_FIXED_CTR2 Ovf 
Set IA32_FIXED_CTR1 Ovf Reserved 
Set IA32_FIXED_CTRO Ovf 


Figure 18-12. IA32_PERF_GLOBAL_STATUS_SET MSR and Architectural Perfmon Version 4 


18.2.4.3 IA32_PERF_GLOBAL_INUSE MSR 


In a contemporary software environment, multiple privileged service agents may wish to employ the processor’s 
performance monitoring facilities. The IA32_MISC_ENABLE.PERFMON_AVAILABLE[bit 7] interface could not serve 
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the need of multiple agent adequately. A white paper, “Performance Monitoring Unit Sharing Guideline 


1 proposed 


a cooperative sharing protocol that is voluntary for participating software agents. 


Architectural performance monitoring version 4 introduces a new MSR, IA32_PERF_GLOBAL_INUSE, that simplifies 
the task of multiple cooperating agents to implement the sharing protocol. 


The layout of IA32_PERF_GLOBAL_INUSE is shown in Figure 18-13. 


63 35 3433 32.31 N... 10 


PMI InUse | 


FIXED_CTR2 InUse 
FIXED_CTR1 InUse PERFEVTSEL(N-1) InUse 


FIXED_CTRO InUse © ——— $$ rrr InUse 
PERFEVTSEL1 InUse 


PERFEVTSELO InUse 


N = CPUID.OAH:EAX[15:8] Reserved 


Figure 18-13. IA32_PERF_GLOBAL_INUSE MSR and Architectural Perfmon Version 4 


The IA32_PERF_GLOBAL_INUSE MSR provides an “InUse” bit for each programmable performance counter and 
fixed counter in the processor. Additionally, it includes an indicator if the PMI mechanism has been configured by a 
profiling agent. 


IA32_PERF_GLOBAL_INUSE.PERFEVTSELO_InUse[bit 0]: This bit reflects the logical state of 
(IA32_PERFEVTSELO[7:0] != 0). 


IA32_PERF_GLOBAL_INUSE.PERFEVTSEL1_InUse[bit 1]: This bit reflects the logical state of 
(IA32_PERFEVTSEL1[7:0] != 0). 


IA32_PERF_GLOBAL_INUSE.PERFEVTSEL2_InUse[bit 2]: This bit reflects the logical state of 
(IA32_PERFEVTSEL2[7:0] != 0). 


IA32_PERF_GLOBAL_INUSE.PERFEVTSELn_InUse[bit n]: This bit reflects the logical state of 
(IA32_PERFEVTSELn[7:0] != 0), n < CPUID.OAH:EAX[15:8]. 


IA32_PERF_GLOBAL_INUSE.FCO_InUse[bit 32]: This bit reflects the logical state of 
(IA32_FIXED_CTR_CTRL[1:0] != 0). 


IA32_PERF_GLOBAL_INUSE.FC1_InUse[bit 33]: This bit reflects the logical state of 
(IA32_FIXED_CTR_CTRL[5:4] != 0). 


IA32_PERF_GLOBAL_INUSE.FC2_InUse[bit 34]: This bit reflects the logical state of 
(IA32_FIXED_CTR_CTRL[9:8] != 0). 


IA32_PERF_GLOBAL_INUSE.PMI_InUse[bit 63]: This bit is set if any one of the following bit is set: 
— JA32_PERFEVTSELn.INT[bit 20], n < CPUID.OAH:EAX[15:8]. 
— JA32_FIXED_CTR_CTRL.ENi_PMI, i = 0, 1, 2. 


— Any IA32_PEBS_ENABLES bit which enables PEBS for a general-purpose or fixed-function performance 
counter. 


1. 


Available at http://www. intel.com/sdm 
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18.2.5 Architectural Performance Monitoring Version 5 


Processors supporting architectural performance monitoring version 5 also support versions 1, 2, 3 and 4, as well 
as capability enumerated by CPUID leaf OAH. Specifically, version 5 provides the following enhancements: 


¢ Deprecation of Anythread mode, see Section 18.2.5.1. 
¢ Individual enumeration of Fixed counters in CPUID.OAH, see Section 18.2.5.2. 
¢ Domain separation, see Section 18.2.5.3. 


18.2.5.1 AnyThread Mode Deprecation 


With Architectural Performance Monitoring Version 5, a processor that supports AnyThread mode deprecation is 
enumerated by CPUID.OAH.EDX[15]. If set, software will not have to follow guidelines in Section 18.2.3.1. 


18.2.5.2 Fixed Counter Enumeration 


With Architectural Performance Monitoring Version 5, register CPUID.OAH.ECX indicates Fixed Counter enumera- 
tion. It is a bit mask which enumerates the supported Fixed Counters in a processor. If bit 'i’ is set, it implies that 
Fixed Counter 'i' is supported. Software is recommended to use the following logic to check if a Fixed Counter is 
supported on a given processor: 


FxCtr[i]_is_ supported := ECX[i] || (EDX[4:0] > i); 


18.2.5.3. Domain Separation 


A counter stops counting when the logical processor exits the CO ACPI C-state. 


18.2.6 — Full-Width Writes to Performance Counter Registers 


The general-purpose performance counter registers IA32_PMCx are writable via WRMSR instruction. However, the 
value written into IA32_PMCx by WRMSR is the signed extended 64-bit value of the EAX[31:0] input of WRMSR. 


A processor that supports full-width writes to the general-purpose performance counters enumerated by 
CPUID.OAH:EAX[15:8] will set IA32_PERF_CAPABILITIES[13] to enumerate its full-width-write capability See 
Figure 18-63. 


If IA32_PERF_CAPABILITIES.FW_WRITE[bit 13] =1, each IA32_PMCi is accompanied by a corresponding alias 
address starting at 4C1H for IA32_A_PMCO. 


The bit width of the performance monitoring counters is specified in CPUID.OAH:EAX[23:16]. 
If IA32_A_PMCi is present, the 64-bit input value (EDX:EAX) of WRMSR to IA32_A_PMCi will cause IA32_PMCi to 
be updated by: 
COUNTERWIDTH = CPUID.OAH:EAX[23:16] bit width of the performance monitoring counter 
IA32_PMCi[COUNTERWIDTH-1:32] := EDX[COUNTERWIDTH-33:0]); 
IA32_PMCi[31:0] := EAX[31:0]; 
EDX[63:COUNTERWIDTH] are reserved 
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18.3. PERFORMANCE MONITORING (INTEL® CORE™ PROCESSORS AND INTEL® 
XEON® PROCESSORS) 


18.3.1. Performance Monitoring for Processors Based on Intel® Microarchitecture Code Name 
Nehalem 


Intel Core i7 processor family? supports architectural performance monitoring capability with version ID 3 (see 
Section 18.2.3) and a host of non-architectural monitoring capabilities. The Intel Core i7 processor family is based 
on Intel® microarchitecture code name Nehalem, and provides four general-purpose performance counters 
(IA32_PMCO, IA32_PMC1, IA32_PMC2, IA32_PMC3) and three fixed-function performance counters 
(IA32_FIXED_CTRO, IA32_FIXED_CTR1, IA32_FIXED_CTR2) in the processor core. 


Non-architectural performance monitoring in Intel Core i7 processor family uses the IA32_PERFEVTSELx MSR to 
configure a set of non-architecture performance monitoring events to be counted by the corresponding general- 
purpose performance counter. The list of non-architectural performance monitoring events can be found at: 
https://perfmon-events.intel.com/. Non-architectural performance monitoring events fall into two broad catego- 
ries: 


¢ Performance monitoring events in the processor core: These include many events that are similar to 
performance monitoring events available to processor based on Intel Core microarchitecture. Additionally, 
there are several enhancements in the performance monitoring capability for detecting microarchitectural 
conditions in the processor core or in the interaction of the processor core to the off-core sub-systems in the 
physical processor package. The off-core sub-systems in the physical processor package is loosely referred to 
as “uncore”. 

¢ Performance monitoring events in the uncore: The uncore sub-system is shared by more than one processor 


cores in the physical processor package. It provides additional performance monitoring facility outside of 
IA32_PMCx and performance monitoring events that are specific to the uncore sub-system. 


Architectural and non-architectural performance monitoring events in Intel Core i7 processor family support thread 
qualification using bit 21 of IA32_PERFEVTSELx MSR. 


The bit fields within each IA32_PERFEVTSELx MSR are defined in Figure 18-6 and described in Section 18.2.1.1 and 
Section 18.2.3. 


63 62 6160 3534333231 876543 21 0 


CHG (RW) 
OVF_PMI (R/W) 


OVF_FC2 (R/O) 
OVF_FC1 (R/O) 
OVF_FCO (R/O) 
OVF_PC7 (R/O), if CCNT>7 
OVF_PC6 (R/O), if CCNT>6 
OVF_PC5 (R/O), if CCNT>5 
OVF_PC4 (R/O), if CCNT>4 
OVF_PC3 (R/O) 


OVF_PC2 (R/O) 
OVF_PC1 (R/O) 
OVF_PCO (R/O) 


Reserved RESET Value — 00000000_00000000H CCNT: CPUID.AH:EAX[15:8] 


Figure 18-14. |A32_PERF_GLOBAL_STATUS MSR 


2. Intel Xeon processor 5500 series and 3400 series are also based on Intel microarchitecture code name Nehalem; the performance 
monitoring facilities described in this section generally also apply. 
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18.3.1.1 Enhancements of Performance Monitoring in the Processor Core 
The notable enhancements in the monitoring of performance events in the processor core include: 


¢ Four general purpose performance counters, IA32_PMCx, associated counter configuration MSRs, 
IA32_PERFEVTSELx, and global counter control MSR supporting simplified control of four counters. Each of the 
four performance counter can support processor event based sampling (PEBS) and thread-qualification of 
architectural and non-architectural performance events. Width of IA32_PMCx supported by hardware has been 
increased. The width of counter reported by CPUID.OAH:EAX[23:16] is 48 bits. The PEBS facility in Intel micro- 
architecture code name Nehalem has been enhanced to include new data format to capture additional infor- 
mation, such as load latency. 


¢ Load latency sampling facility. Average latency of memory load operation can be sampled using load-latency 
facility in processors based on Intel microarchitecture code name Nehalem. This field measures the load 
latency from load's first dispatch of till final data writeback from the memory subsystem. The latency is 
reported for retired demand load operations and in core cycles (it accounts for re-dispatches). This facility is 
used in conjunction with the PEBS facility. 


°  Off-core response counting facility. This facility in the processor core allows software to count certain 
transaction responses between the processor core to sub-systems outside the processor core (uncore). 
Counting off-core response requires additional event qualification configuration facility in conjunction with 
IA32_PERFEVTSELx. Two off-core response MSRs are provided to use in conjunction with specific event codes 
that must be specified with IA32_PERFEVTSELx. 


NOTE 


The number of counters available to software may vary from the number of physical counters 
present on the hardware, because an agent running at a higher privilege level (e.g., a VMM) may 
not expose all counters. CPUID.OAH:EAX[15:8] reports the MSRs available to software; see Section 
18.2.1. 


18.3.1.1.1 Processor Event Based Sampling (PEBS) 


All general-purpose performance counters, IA32_PMCx, can be used for PEBS if the performance event supports 
PEBS. Software uses IA32_MISC_ENABLE[7] and IA32_MISC_ENABLE[12] to detect whether the performance 
monitoring facility and PEBS functionality are supported in the processor. The MSR IA32_PEBS_ENABLE provides 4 
bits that software must use to enable which IA32_PMCx overflow condition will cause the PEBS record to be 
captured. 


Additionally, the PEBS record is expanded to allow latency information to be captured. The MSR 
IA32_PEBS_ENABLE provides 4 additional bits that software must use to enable latency data recording in the PEBS 
record upon the respective IA32_PMCx overflow condition. The layout of IA32_PEBS_ENABLE for processors based 
on Intel microarchitecture code name Nehalem is shown in Figure 18-15. 


When a counter is enabled to capture machine state (PEBS_EN_PMCx = 1), the processor will write machine state 
information to a memory buffer specified by software as detailed below. When the counter IA32_PMCx overflows 
from maximum count to zero, the PEBS hardware is armed. 
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63 36 3534 33 32 31 87654321 0 


LL_EN_PMC3 (RW) 
LL_EN_PMC2 (R/W) 
LL_EN_PMC1 (RW) 
LL_EN_PMCO (R/W) 
PEBS_EN_PMC3 (R/W) 
PEBS_EN_ PMC2 (R/W) 
PEBS_EN PMC1 (R/W) 
PEBS_EN_ PMCO (R/W) 


Reserved RESET Value — 00000000_00000000H 


Figure 18-15. Layout of IA32_PEBS_ENABLE MSR 


Upon occurrence of the next PEBS event, the PEBS hardware triggers an assist and causes a PEBS record to be 
written. The format of the PEBS record is indicated by the bit field IA32_PERF_CAPABILITIES[11:8] (see 

Figure 18-63). 

The behavior of PEBS assists is reported by IA32_PERF_CAPABILITIES[6] (see Figure 18-63). The return instruc- 
tion pointer (RIP) reported in the PEBS record will point to the instruction after (+1) the instruction that causes the 
PEBS assist. The machine state reported in the PEBS record is the machine state after the instruction that causes 
the PEBS assist is retired. For instance, if the instructions: 


mov eax, [eax] ; causes PEBS assist 
nop 


are executed, the PEBS record will report the address of the nop, and the value of EAX in the PEBS record will show 
the value read from memory, not the target address of the read operation. 


The PEBS record format is shown in Table 18-3, and each field in the PEBS record is 64 bits long. The PEBS record 
format, along with debug/store area storage format, does not change regardless of IA-32e mode is active or not. 
CPUID.01H:ECX.DTES64[bit 2] reports whether the processor's DS storage format support is mode-independent. 
When set, it uses 64-bit DS storage format. 


Table 18-3. PEBS Record Format for Intel Core i7 Processor Family 


Byte Offset Field Byte Offset Field 

OOH R/EFLAGS 58H RI 

08H R/EIP 60H R10 

10H R/EAX 68H R11 

18H R/EBX 70H R12 

20H R/ECX 78H R13 

28H R/EDX 80H R14 

30H R/ESI 88H R15 

38H R/EDI 90H IA32_PERF_GLOBAL_STATUS 
40H R/EBP 98H Data Linear Address 

48H R/ESP AOH Data Source Encoding 

50H R8 A8H Latency value (core cycles) 
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In IA-32e mode, the full 64-bit value is written to the register. If the processor is not operating in IA-32e mode, 32- 
bit value is written to registers with bits 63:32 zeroed. Registers not defined when the processor is not in IA-32e 
mode are written to zero. 


Bytes AFH:90H are enhancement to the PEBS record format. Support for this enhanced PEBS record format is indi- 
cated by IA32_PERF_CAPABILITIES[11:8] encoding of 0001B. 


The value written to bytes 97H:90H is the state of the IA32_PERF_GLOBAL_STATUS register before the PEBS assist 
occurred. This value is written so software can determine which counters overflowed when this PEBS record was 
written. Note that this field indicates the overflow status for all counters, regardless of whether they were 
programmed for PEBS or not. 


Programming PEBS Facility 


Only a subset of non-architectural performance events in the processor support PEBS. The subset of precise events 
are listed in Table 18-78. In addition to using IA32_PERFEVTSELx to specify event unit/mask settings and setting 

the EN_PMCx bit in the IA32_PEBS_ENABLE register for the respective counter, the software must also initialize the 
DS_BUFFER_MANAGEMENT_AREA data structure in memory to support capturing PEBS records for precise events. 


The recording of PEBS records may not operate properly if accesses to the linear addresses in the DS buffer 
management area or in the PEBS buffer (see below) cause page faults, VM exits, or the setting of accessed or dirty 
flags in the paging structures (ordinary or EPT). For that reason, system software should establish paging struc- 
tures (both ordinary and EPT) to prevent such occurrences. Implications of this may be that an operating system 
should allocate this memory from a non-paged pool and that system software cannot do “lazy” page-table entry 
propagation for these pages. A virtual-machine monitor may choose to allow use of PEBS by guest software only if 
EPT maps all guest-physical memory as present and read/write. 


NOTE 


PEBS events are only valid when the following fields of IA32_PERFEVTSELx are all zero: AnyThread, 
Edge, Invert, CMask. 


The beginning linear address of the DS_BUFFER_MANAGEMENT_AREA data structure must be programmed into 
the IA32_DS_AREA register. The layout of the DS_BUFFER_MANAGEMENT_AREA is shown in Figure 18-16. 


¢ PEBS Buffer Base: This field is programmed with the linear address of the first byte of the PEBS buffer 
allocated by software. The processor reads this field to determine the base address of the PEBS buffer. 


¢ PEBS Index: This field is initially programmed with the same value as the PEBS Buffer Base field, or the 
beginning linear address of the PEBS buffer. The processor reads this field to determine the location of the next 
PEBS record to write to. After a PEBS record has been written, the processor also updates this field with the 
address of the next PEBS record to be written. The figure above illustrates the state of PEBS Index after the first 
PEBS record is written. 


¢ PEBS Absolute Maximum: This field represents the absolute address of the maximum length of the allocated 
PEBS buffer plus the starting address of the PEBS buffer. The processor will not write any PEBS record beyond 
the end of PEBS buffer, when PEBS Index equals PEBS Absolute Maximum. No signaling is generated when 
PEBS buffer is full. Software must reset the PEBS Index field to the beginning of the PEBS buffer address to 
continue capturing PEBS records. 


Vol. 3B 18-21 


PERFORMANCE MONITORING 


-— IA32_DS_AREA MSR 


DS Buffer Management Area BTS Buffer 
\<<—_ 
BTS Buffer Base | OH———————> 
Branch Record 0 
BTS Index 8H 
BTS Absolute 
Maximum 10 
BTS Interrupt Branch Record 1 
Threshold 18H 
PEBS Buffer Base) 20H 
PEBS Index 28H —q 
serv Absolute | 35), 
jaximum 
-———_—_______, Branch Record n 
PEBS Interrupt | 33H 
Threshold $3 
PEBS 40H 
CounterO Reset PEBS Buffer 
48H 
PEBS —_— 
Counter1 Reset PEBS Record 0 
50H 
PEBS 
Counter2 Reset 
PEBS Record 1 
58H 
PEBS 
Counter3 Reset | 
Reserved oun 
PEBS Record n 
> 


Figure 18-16. PEBS Programming Environment 


PEBS Interrupt Threshold: This field specifies the threshold value to trigger a performance interrupt and 
notify software that the PEBS buffer is nearly full. This field is programmed with the linear address of the first 
byte of the PEBS record within the PEBS buffer that represents the threshold record. After the processor writes 
a PEBS record and updates PEBS Index, if the PEBS Index reaches the threshold value of this field, the 
processor will generate a performance interrupt. This is the same interrupt that is generated by a performance 
counter overflow, as programmed in the Performance Monitoring Counters vector in the Local Vector Table of 
the Local APIC. When a performance interrupt due to PEBS buffer full is generated, the 
IA32_PERF_GLOBAL_STATUS.PEBS_Ovf bit will be set. 


PEBS CounterX Reset: This field allows software to set up PEBS counter overflow condition to occur at a rate 
useful for profiling workload, thereby generating multiple PEBS records to facilitate characterizing the profile 
the execution of test code. After each PEBS record is written, the processor checks each counter to see if it 
overflowed and was enabled for PEBS (the corresponding bit in IA32_PEBS_ENABLED was set). If these 
conditions are met, then the reset value for each overflowed counter is loaded from the DS Buffer Management 
Area. For example, if counter IA32_PMCO caused a PEBS record to be written, then the value of “PEBS Counter 
0 Reset” would be written to counter IA32_PMCO. If a counter is not enabled for PEBS, its value will not be 
modified by the PEBS assist. 


Performance Counter Prioritization 


Performance monitoring interrupts are triggered by a counter transitioning from maximum count to zero (assuming 
IA32_PerfEvtSelX.INT is set). This same transition will cause PEBS hardware to arm, but not trigger. PEBS hard- 
ware triggers upon detection of the first PEBS event after the PEBS hardware has been armed (a 0 to 1 transition 
of the counter). At this point, a PEBS assist will be undertaken by the processor. 
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Performance counters (fixed and general-purpose) are prioritized in index order. That is, counter IA32_PMCO takes 
precedence over all other counters. Counter IA32_PMC1 takes precedence over counters IA32_PMC2 and 
IA32_PMC3, and so on. This means that if simultaneous overflows or PEBS assists occur, the appropriate action will 
be taken for the highest priority performance counter. For example, if IA32_PMC1i cause an overflow interrupt and 
IA32_PMC2 causes an PEBS assist simultaneously, then the overflow interrupt will be serviced first. 


The PEBS threshold interrupt is triggered by the PEBS assist, and is by definition prioritized lower than the PEBS 
assist. Hardware will not generate separate interrupts for each counter that simultaneously overflows. General- 
purpose performance counters are prioritized over fixed counters. 


If a counter is programmed with a precise (PEBS-enabled) event and programmed to generate a counter overflow 
interrupt, the PEBS assist is serviced before the counter overflow interrupt is serviced. If in addition the PEBS inter- 
rupt threshold is met, the 


threshold interrupt is generated after the PEBS assist completes, followed by the counter overflow interrupt (two 
separate interrupts are generated). 


Uncore counters may be programmed to interrupt one or more processor cores (see Section 18.3.1.2). It is 
possible for interrupts posted from the uncore facility to occur coincident with counter overflow interrupts from the 
processor core. Software must check core and uncore status registers to determine the exact origin of counter 
overflow interrupts. 


18.3.1.1.2 Load Latency Performance Monitoring Facility 


The load latency facility provides software a means to characterize the average load latency to different levels of 
cache/memory hierarchy. This facility requires processor supporting enhanced PEBS record format in the PEBS 
buffer, see Table 18-3. This field measures the load latency from load's first dispatch of till final data writeback from 
the memory subsystem. The latency is reported for retired demand load operations and in core cycles (it accounts 
for re-dispatches). 


To use this feature software must assure: 


° One of the IA32_PERFEVTSELx MSR is programmed to specify the event unit MEM_INST_RETIRED, and the 
LATENCY_ABOVE_THRESHOLD event mask must be specified (IA32_PerfEvtSelX[15:0] = 100H). The corre- 
sponding counter IA32_PMCx will accumulate event counts for architecturally visible loads which exceed the 
programmed latency threshold specified separately in a MSR. Stores are ignored when this event is 
programmed. The CMASK or INV fields of the IA32_PerfEvtSelX register used for counting load latency must be 
0. Writing other values will result in undefined behavior. 


¢ The MSR_PEBS_LD_LAT_THRESHOLD MSR is programmed with the desired latency threshold in core clock 
cycles. Loads with latencies greater than this value are eligible for counting and latency data reporting. The 
minimum value that may be programmed in this register is 3 (the minimum detectable load latency is 4 core 
clock cycles). 


° The PEBS enable bit in the IA32_PEBS_ENABLE register is set for the corresponding IA32_PMCx counter 
register. This means that both the PEBS_EN_CTRX and LL_EN_CTRX bits must be set for the counter(s) of 
interest. For example, to enable load latency on counter IA32_PMCO, the IA32_PEBS_ENABLE register must be 
programmed with the 64-bit value 00000001_O00000001H. 


When the load-latency facility is enabled, load operations are randomly selected by hardware and tagged to carry 
information related to data source locality and latency. Latency and data source information of tagged loads are 
updated internally. 


When a PEBS assist occurs, the last update of latency and data source information are captured by the assist and 
written as part of the PEBS record. The PEBS sample after value (SAV), specified in PEBS CounterX Reset, operates 
orthogonally to the tagging mechanism. Loads are randomly tagged to collect latency data. The SAV controls the 
number of tagged loads with latency information that will be written into the PEBS record field by the PEBS assists. 
The load latency data written to the PEBS record will be for the last tagged load operation which retired just before 
the PEBS assist was invoked. 


The load-latency information written into a PEBS record (see Table 18-3, bytes AFH:98H) consists of: 
° Data Linear Address: This is the linear address of the target of the load operation. 


¢ Latency Value: This is the elapsed cycles of the tagged load operation between dispatch to GO, measured in 
processor core clock domain. 
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¢ Data Source: The encoded value indicates the origin of the data obtained by the load instruction. The encoding 
is shown in Table 18-4. In the descriptions, local memory refers to system memory physically attached to a 
processor package, and remote memory refers to system memory physically attached to another processor 
package. 


Table 18-4. Data Source Encoding for Load Latency Record 


Encoding Description 

OOH Unknown L3 cache miss. 

01H Minimal latency core cache hit. This request was satisfied by the L1 data cache. 

02H Pending core cache HIT. Outstanding core cache miss to same cache-line address was already underway. 

03H This data request was satisfied by the L2. 

04H L3 HIT. Local or Remote home requests that hit L3 cache in the uncore with no coherency actions required (snooping). 
05H L3 HIT. Local or Remote home requests that hit the L3 cache and were serviced by another processor core with a 


cross core snoop where no modified copies were found. (clean). 


O6H L3 HIT. Local or Remote home requests that hit the L3 cache and were serviced by another processor core with a 
cross core snoop where no modified copies were found. 


07H! Reserved/LLC Snoop HitM. Local or Remote home requests that hit the last level cache and were serviced by another 
core with a cross core snoop where modified copies were found. 


08H Reserved/L3 MISS. Local homed requests that missed the L3 cache and were serviced by forwarded data following a 
cross package snoop where no modified copies were found. (Remote home requests are not counted). 


09H Reserved 


OAH L3 MISS. Local home requests that missed the L3 cache and were serviced by local DRAM (go to shared state). 

OBH L3 MISS. Remote home requests that missed the L3 cache and were serviced by remote DRAM (go to shared state). 
OCH L3 MISS. Local home requests that missed the L3 cache and were serviced by local DRAM (go to exclusive state). 
ODH L3 MISS. Remote home requests that missed the L3 cache and were serviced by remote DRAM (go to exclusive state). 
OEH I/O, Request of input/output operation. 

OFH The request was to un-cacheable memory. 

NOTES: 


1. Bit 7 is supported only for processors with a CPUID DisplayFamily_DisplayModel signature of O6_2A, and 06_2E; otherwise it is 
reserved. 


The layout of MSR_PEBS_LD_LAT_THRESHOLD is shown in Figure 18-17. 


63 1615 0 


THRHLD - Load latency threshold 


Reserved RESET Value — 00000000_00000000H 


Figure 18-17. Layout of MSR_PEBS_LD_LAT MSR 


Bits 15:0 specifies the threshold load latency in core clock cycles. Performance events with latencies greater than 
this value are counted in IA32_PMCx and their latency information is reported in the PEBS record. Otherwise, they 
are ignored. The minimum value that may be programmed in this field is 3. 
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18.3.1.1.3 Off-core Response Performance Monitoring in the Processor Core 


Programming a performance event using the off-core response facility can choose any of the four 
IA32_PERFEVTSELx MSR with specific event codes and predefine mask bit value. Each event code for off-core 
response monitoring requires programming an associated configuration MSR, MSR_OFFCORE_RSP_O. There is 
only one off-core response configuration MSR. Table 18-5 lists the event code, mask value and additional off-core 
configuration MSR that must be programmed to count off-core response events using IA32_PMCx. 


Table 18-5. Off-Core Response Event Encoding 


Event code in Mask Value in 
IA32_PERFEVTSELx IA32_PERFEVTSELx Required Off-core Response MSR 
B7H 01H MSR_OFFCORE_RSP_O (address 1A6H) 


The layout of MSR_OFFCORE_RSP_0O is shown in Figure 18-18. Bits 7:0 specifies the request type of a transaction 
request to the uncore. Bits 15:8 specifies the response of the uncore subsystem. 


63 15 1413121110 9876 543 21 0 


RESPONSE TYPE — NON_DRAM (R/W) | 
RESPONSE TYPE — LOCAL_DRAM (R/W) 


RESPONSE TYPE — REMOTE_DRAM (R/W) 
RESPONSE TYPE — REMOTE_CACHE_FWD (R/W) 
RESPONSE TYPE — RESERVED —— 
RESPONSE TYPE — OTHER_CORE_HITM (R/W) 

RESPONSE TYPE — OTHER_CORE_HIT_SNP (R/W) 
RESPONSE TYPE — UNCORE_HIT (R/W) 
REQUEST TYPE — OTHER (R/W) 
REQUEST TYPE — PF_IFETCH (R/W) 
REQUEST TYPE — PF_RFO (R/W) 
REQUEST TYPE — PF_DATA_RD (R/W) 
REQUEST TYPE — WB (R/W) 
REQUEST TYPE — DMND_IFETCH (R/W) 
REQUEST TYPE — DMND_RFO (R/W) 
REQUEST TYPE — DMND_DATA RD (R/W) 


Reserved RESET Value — 00000000_00000000H 


Figure 18-18. Layout of MSR_OFFCORE_RSP_O and MSR_OFFCORE_RSP_1 to Configure Off-core Response Events 


Table 18-6. MSR_OFFCORE_RSP_O and MSR_OFFCORE_RSP_1 Bit Field Definition 
Bit Name Offset Description 


DMND_DATA_RD | 0 Counts the number of demand and DCU prefetch data reads of full and partial cachelines as well as 
demand data page table entry cacheline reads. Does not count L2 data read prefetches or instruction 
fetches. 

DMND_RFO 1 Counts the number of demand and DCU prefetch reads for ownership (RFO) requests generated by a 


write to data cacheline. Does not count L2 RFO. 


DMND_IFETCH 2 Counts the number of demand instruction cacheline reads and L1 instruction cacheline prefetches. 
WB 3 Counts the number of writeback (modified to exclusive) transactions. 

PF_DATA_RD 4 Counts the number of data cacheline reads generated by L2 prefetchers. 

PF_RFO 5 Counts the number of RFO requests generated by L2 prefetchers. 

PF_IFETCH 6 Counts the number of code reads generated by L2 prefetchers. 
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Table 18-6. MSR_OFFCORE_RSP_O and MSR_OFFCORE_RSP_1 Bit Field Definition (Contd.) 


Bit Name Offset Description 

OTHER 7 Counts one of the following transaction types, including L3 invalidate, I/O, full or partial writes, WC or 
non-temporal stores, CLFLUSH, Fences, lock, unlock, split lock. 

UNCORE_HIT 8 L3 Hit: local or remote home requests that hit L3 cache in the uncore with no coherency actions 
required (Snooping). 

OTHER_CORE_HI | 9 L3 Hit: local or remote home requests that hit L3 cache in the uncore and was serviced by another 

T_SNP core with a cross core snoop where no modified copies were found (clean). 

OTHER_CORE_HI | 10 L3 Hit: local or remote home requests that hit L3 cache in the uncore and was serviced by another 

™ core with a cross core snoop where modified copies were found (HITM). 

Reserved 11 Reserved 

REMOTE_CACHE_| 12 L3 Miss: local homed requests that missed the L3 cache and was serviced by forwarded data following 

FWD a cross package snoop where no modified copies found. (Remote home requests are not counted) 

REMOTE_DRAM | 13 L3 Miss: remote home requests that missed the L3 cache and were serviced by remote DRAM. 

LOCAL_DRAM 14 L3 Miss: local home requests that missed the L3 cache and were serviced by local DRAM. 

NON_DRAM 15 Non-DRAM requests that were serviced by IOH. 


18.3.1.2 Performance Monitoring Facility in the Uncore 


The “uncore” in Intel microarchitecture code name Nehalem refers to subsystems in the physical processor 
package that are shared by multiple processor cores. Some of the sub-systems in the uncore include the L3 cache, 
Intel QuickPath Interconnect link logic, and integrated memory controller. The performance monitoring facilities 
inside the uncore operates in the same clock domain as the uncore (U-clock domain), which is usually different 
from the processor core clock domain. The uncore performance monitoring facilities described in this section apply 
to Intel Xeon processor 5500 series and processors with the following CPUID signatures: 06_1AH, 06_1EH, 06_1FH 
(see Chapter 2, “Model-Specific Registers (MSRs)” in the Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4). An overview of the uncore performance monitoring facilities is described separately. 


The performance monitoring facilities available in the U-clock domain consist of: 


° Eight General-purpose counters (MSR_UNCORE_PerfCntr0 through MSR_UNCORE_PerfCntr7). The counters 
are 48 bits wide. Each counter is associated with a configuration MSR, MSR_UNCORE_PerfEvtSelx, to specify 
event code, event mask and other event qualification fields. A set of global uncore performance counter 
enabling/overflow/status control MSRs are also provided for software. 

¢ Performance monitoring in the uncore provides an address/opcode match MSR that provides event qualification 
control based on address value or QPI command opcode. 

© One fixed-function counter, MSR_UNCORE_FixedCntr0. The fixed-function uncore counter increments at the 
rate of the U-clock when enabled. 

The frequency of the uncore clock domain can be determined from the uncore clock ratio which is available in 
the PCI configuration space register at offset COH under device number 0 and Function 0. 


18.3.1.2.1_ Uncore Performance Monitoring Management Facility 


MSR_UNCORE_PERF_GLOBAL_CTRL provides bit fields to enable/disable general-purpose and fixed-function coun- 
ters in the uncore. Figure 18-19 shows the layout of MSR_UNCORE_PERF_GLOBAL_CTRL for an uncore that is 
shared by four processor cores in a physical package. 


¢ EN_PCn (bit n, n = 0, 7): When set, enables counting for the general-purpose uncore counter 
MSR_UNCORE_PerfCntr n. 


¢ EN_FCO (bit 32): When set, enables counting for the fixed-function uncore counter MSR_UNCORE_FixedCntr0. 


¢ EN_PMI_COREn (bit n, n = 0, 3 if four cores are present): When set, processor core n is programmed to receive 
an interrupt signal from any interrupt enabled uncore counter. PMI delivery due to an uncore counter overflow 
is enabled by setting IA32_DEBUGCTL.Offcore_PMI_EN to 1. 
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¢ =PMI_FRZ (bit 63): When set, all U-clock uncore counters are disabled when any one of them signals a 


performance interrupt. Software must explicitly re-enable the counter by setting the enable bits in 
MSR_UNCORE_PERF_GLOBAL_CTRL upon exit from the ISR. 


63 62 51 50 49 48 


32 31 


876 543 21 


0 


PMI_FRZ (R/W) 
EN_PMI_CORE3 (R/W 
EN_PMI_CORE2 (RW) — 


EN_PMI_CORE1 (RW) —— 
EN_PMI_COREO (R/W) 
EN_FCO (RW) 


EN_PC7 (R/W) 


EN_PC6 (R/W) 
WwW 


EN_PC4 (R/W) 


R 
EN_PC5 (R/W) 
R 


EN_PC2 (R/W) 


EN _PC1 (R/W) 


( 

( 

( 
EN_PC3 (R/W) 

( 

( 

( 


EN_PCO (R/W) 


Reserved 


RESET Value — 00000000_00000000H 


Figure 18-19. Layout of MSR_UNCORE_PERF_GLOBAL_CTRL MSR 


MSR_UNCORE_PERF_GLOBAL_STATUS provides overflow status of the U-clock performance counters in the 
uncore. This is a read-only register. If an overflow status bit is set the corresponding counter has overflowed. The 
register provides a condition change bit (bit 63) which can be quickly checked by software to determine if a signif- 
icant change has occurred since the last time the condition change status was cleared. Figure 18-20 shows the 


layout of MSR_UNCORE_PERF_GLOBAL_STATUS. 


© OVF_PCn (bit n, n = 0, 7): When set, indicates general-purpose uncore counter MSR_UNCORE_PerfCntr n has 


overflowed. 


® OVF_FCO (bit 32): When set, indicates the fixed-function uncore counter MSR_UNCORE_FixedCntr0 has 


overflowed. 


° OVF_PMI (bit 61): When set indicates that an uncore counter overflowed and generated an interrupt request. 
° CHG (bit 63): When set indicates that at least one status bit in MSR_UNCORE_PERF_GLOBAL_STATUS register 


has changed state. 
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MSR_UNCORE_PERF_GLOBAL_OVF_CTRL allows software to clear the status bits in the 
UNCORE_PERF_GLOBAL_STATUS register. This is a write-only register, and individual status bits in the global 
status register are cleared by writing a binary one to the corresponding bit in this register. Writing zero to any bit 
position in this register has no effect on the uncore PMU hardware. 


63 62 6160 32 31 876543 21 0 


| 

CHG (RW) | 
OVF_PMI (RW 

OVF_FCO (R/O) 
OVF_PC7 (R/O) 
OVF_PC6 (R/O) 
OVF_PC5 (R/O) 
OVF_PC4 (R/O) 
OVF_PC3 (R/O) 
OVF_PC2 (R/O) 
OVF_PC1 (R/O) 
OVF_PCO (R/O) 


Reserved RESET Value — 00000000_00000000H 


Figure 18-20. Layout of MSR_UNCORE_PERF_GLOBAL_STATUS MSR 


Figure 18-21 shows the layout of MSR_UNCORE_PERF_GLOBAL_OVF_CTRL. 


63 62 6160 32 31 87654321 0 


‘ion 
CLR_CHG (WO1) 
CLR_OVF_PMI (WO1) 
CLR_OVF_FCO (WO1) 
CLR_OVF_PC7 (WO1) 
CLR_OVF_PC6 (WO1) 
CLR_OVF_PC5 (WO1) 
CLR_OVF_PC4 (WO1) 
CLR_OVF_PC3 (WO1) 
CLR_OVF_PC2 (WO1) 
CLR_OVF_PC1 (WO1) 
CLR_OVF_PCO (WO1) 


Reserved RESET Value — 00000000_00000000H 


Figure 18-21. Layout of MSR_UNCORE_PERF_GLOBAL_OVF_CTRL MSR 


¢ CLR_OVF_PCn (bit n, n = 0, 7): Set this bit to clear the overflow status for general-purpose uncore counter 
MSR_UNCORE_PerfCntr n. Writing a value other than 1 is ignored. 


¢ CLR_OVF_FCO (bit 32): Set this bit to clear the overflow status for the fixed-function uncore counter 
MSR_UNCORE_FixedCntr0. Writing a value other than 1 is ignored. 


¢ CLR_OVF_PMI (bit 61): Set this bit to clear the OVF_PMI flag in MSR_UNCORE_PERF_GLOBAL_STATUS. Writing 
a value other than 1 is ignored. 


® CLR_CHG (bit 63): Set this bit to clear the CHG flag in MSR_UNCORE_PERF_GLOBAL_STATUS register. Writing 
a value other than 1 is ignored. 
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18.3.1.2.2 Uncore Performance Event Configuration Facility 


MSR_UNCORE_PerfEvtSel0 through MSR_UNCORE_PerfEvtSel7 are used to select performance event and 
configure the counting behavior of the respective uncore performance counter. Each uncore PerfEvtSel MSR is 
paired with an uncore performance counter. Each uncore counter must be locally configured using the corre- 
sponding MSR_UNCORE_PerfEvtSelx and counting must be enabled using the respective EN_PCx bit in 
MSR_UNCORE_PERF_GLOBAL_CTRL. Figure 18-22 shows the layout of MSR_UNCORE_PERFEVTSELx. 


63 31 24 23 22 21 201918171615 8 7 0 


art Unit Mask (UMASK)| —_ Event Select 


INV—Invert counter mask ——! 


EN—Enable counters ————_! Reserved 
PMI—Enable PMI on overflow ———__ 
E—Edge detect RESET Value — 00000000_00000000H 


OCC_CTR_RST—Rest Queue Occ 


Figure 18-22. Layout of MSR_UNCORE_PERFEVTSELx MSRs 


¢ Event Select (bits 7:0): Selects the event logic unit used to detect uncore events. 
¢ Unit Mask (bits 15:8) : Condition qualifiers for the event selection logic specified in the Event Select field. 


* OCC_CTR_RST (bit17): When set causes the queue occupancy counter associated with this event to be cleared 
(zeroed). Writing a zero to this bit will be ignored. It will always read as a zero. 


° Edge Detect (bit 18): When set causes the counter to increment when a deasserted to asserted transition 
occurs for the conditions that can be expressed by any of the fields in this register. 


¢ PMI (bit 20): When set, the uncore will generate an interrupt request when this counter overflowed. This 
request will be routed to the logical processors as enabled in the PMI enable bits (EN_PMI_COREx) in the 
register MSR_UNCORE_PERF_GLOBAL_CTRL. 


e EN (bit 22): When clear, this counter is locally disabled. When set, this counter is locally enabled and counting 
starts when the corresponding EN_PCx bit in MSR_UNCORE_PERF_GLOBAL_CTRL is set. 


e INV (bit 23): When clear, the Counter Mask field is interpreted as greater than or equal to. When set, the 
Counter Mask field is interpreted as less than. 


* Counter Mask (bits 31:24): When this field is clear, it has no effect on counting. When set to a value other than 
zero, the logical processor compares this field to the event counts on each core clock cycle. If INV is clear and 
the event counts are greater than or equal to this field, the counter is incremented by one. If INV is set and the 
event counts are less than this field, the counter is incremented by one. Otherwise the counter is not incre- 
mented. 


Figure 18-23 shows the layout of MSR_UNCORE_FIXED_CTR_CTRL. 


63 87654321 0 


PMI - Generate PMI on overflow | 
EN - Enable 


Reserved RESET Value — 00000000_00000000H 


Figure 18-23. Layout of MSR_UNCORE_FIXED_CTR_CTRL MSR 
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° EN (bit 0): When clear, the uncore fixed-function counter is locally disabled. When set, it is locally enabled and 
counting starts when the EN_FCO bit in MSR_UNCORE_PERF_GLOBAL_CTRL is set. 


¢ PMI (bit 2): When set, the uncore will generate an interrupt request when the uncore fixed-function counter 
overflowed. This request will be routed to the logical processors as enabled in the PMI enable bits 
(EN_PMI_COREx) in the register MSR_UNCORE_PERF_GLOBAL_CTRL. 


Both the general-purpose counters (MSR_UNCORE_PerfCntr) and the fixed-function counter 
(MSR_UNCORE_FixedCntr0) are 48 bits wide. They support both counting and interrupt based sampling usages. 
The event logic unit can filter event counts to specific regions of code or transaction types incoming to the home 
node logic. 


18.3.1.2.3. Uncore Address/Opcode Match MSR 


The Event Select field [7:0] of MSR_UNCORE_PERFEVTSELx is used to select different uncore event logic unit. 
When the event “ADDR_OPCODE_MATCH"” is selected in the Event Select field, software can filter uncore perfor- 
mance events according to transaction address and certain transaction responses. The address filter and transac- 
tion response filtering requires the use of MSR_UNCORE_ADDR_OPCODE_MATCH register. The layout is shown in 
Figure 18-24. 


63 60 4847 4039 320 


Opcode ADDR 


MatchSel—Select addr/Opcode 
Opcode—Opcode and Message 
ADDR—Bits 39:4 of physical address 


Reserved RESET Value — 00000000_00000000H 


Figure 18-24. Layout of MSR_UNCORE_ADDR_OPCODE_MATCH MSR 


¢ Addr (bits 39:3): The physical address to match if “MatchSel" field is set to select address match. The uncore 
performance counter will increment if the lowest 40-bit incoming physical address (excluding bits 2:0) for a 
transaction request matches bits 39:3. 


® Opcode (bits 47:40) : Bits 47:40 allow software to filter uncore transactions based on QPI link message 
class/packed header opcode. These bits are consists two sub-fields: 


— Bits 43:40 specify the QPI packet header opcode. 
— Bits 47:44 specify the QPI message classes. 
Table 18-7 lists the encodings supported in the opcode field. 
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Table 18-7. Opcode Field Encoding for MSR_UNCORE_ADDR_OPCODE_MATCH 


Opcode [43:40] QP! Message Class 
Home Request Snoop Response Data Response 
[47:44] = 0O00B [47:44] = 0001B [47:44] = 1110B 

1 

DMND_IFETCH 2 2 

WB 3 3 

PF_DATA_RD 4 4 

PF_RFO 5 5 

PF_IFETCH 6 6 

OTHER 7 7 

NON_DRAM 15 15 


¢ MatchSel (bits 63:61): Software specifies the match criteria according to the following encoding: 
— OOOB: Disable addr_opcode match hardware. 
— 100B: Count if only the address field matches. 
— 010B: Count if only the opcode field matches. 
— 110B: Count if either opcode field matches or the address field matches. 
— 001B: Count only if both opcode and address field match. 
— Other encoding are reserved. 


18.3.1.3 Intel® Xeon® Processor 7500 Series Performance Monitoring Facility 


The performance monitoring facility in the processor core of Intel® Xeon® processor 7500 series are the same as 
those supported in Intel Xeon processor 5500 series. The uncore subsystem in Intel Xeon processor 7500 series 
are significantly different The uncore performance monitoring facility consist of many distributed units associated 
with individual logic control units (referred to as boxes) within the uncore subsystem. A high level block diagram of 
the various box units of the uncore is shown in Figure 18-25. 


Uncore PMUs are programmed via MSR interfaces. Each of the distributed uncore PMU units have several general- 
purpose counters. Each counter requires an associated event select MSR, and may require additional MSRs to 
configure sub-event conditions. The uncore PMU MSRs associated with each box can be categorized based on its 
functional scope: per-counter, per-box, or global across the uncore. The number counters available in each box 
type are different. Each box generally provides a set of MSRs to enable/disable, check status/overflow of multiple 
counters within each box. 
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CBox CBox CBox CBox CBox CBox CBox CBox 


SBox SBox 
SMI Channels 
: - PBox MBox BBox RBox BBox MBox PBox ane 
SMI Channels 
WBox PBox PBox PBox PBox UBox 


oe, 


4 Intel QPI Links 


Figure 18-25. Distributed Units of the Uncore of Intel® Xeon® Processor 7500 Series 


Table 18-8 summarizes the number MSRs for uncore PMU for each box. 


Table 18-8. Uncore PMU MSR Summary 


Counter General Global 

Box # of Boxes | Counters per Box Width Purpose Enable Sub-control MSRs 
C-Box 8 6 48 Yes per-box None 
S-Box 2 4 48 Yes per-box Match/Mask 
B-Box 2 4 48 Yes per-box Match/Mask 
M-Box 2 6 48 Yes per-box Yes 
R-Box 1 16 (2 port, 8 per port) 48 Yes per-box Yes 
W-Box 1 4 48 Yes per-box None 

1 48 No per-box None 
U-Box 1 1 48 Yes uncore None 


The W-Box provides 4 general-purpose counters, each requiring an event select configuration MSR, similar to the 
general-purpose counters in other boxes. There is also a fixed-function counter that increments clockticks in the 
uncore clock domain. 


For C,S,B,M,R, and W boxes, each box provides an MSR to enable/disable counting, configuring PMI of multiple 
counters within the same box, this is somewhat similar the “global control“ programming interface, 
IA32_PERF_GLOBAL_CTRL, offered in the core PMU. Similarly status information and counter overflow control for 
multiple counters within the same box are also provided in C,S,B,M,R, and W boxes. 


In the U-Box, MSR_U_PMON_GLOBAL_CTL provides overall uncore PMU enable/disable and PMI configuration 
control. The scope of status information in the U-box is at per-box granularity, in contrast to the per-box status 
information MSR (in the C,S,B,M,R, and W boxes) providing status information of individual counter overflow. The 
difference in scope also apply to the overflow control MSR in the U-Box versus those in the other Boxes. 
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The individual MSRs that provide uncore PMU interfaces are listed in Chapter 2, “Model-Specific Registers (MSRs)” 
in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4, Table 2-17 under the general 
naming style of MSR_%box#%_PMON_%scope_function%, where %box#% designates the type of box and zero- 
based index if there are more the one box of the same type, %scope_function% follows the examples below: 


¢ =Multi-counter enabling MSRs: MSR_U_PMON_GLOBAL_CTL, MSR_SO_PMON_BOX_CTL, 
MSR_C7_PMON_BOX_CTL, etc. 


¢ Multi-counter status MSRs: MSR_U_PMON_GLOBAL_STATUS, MSR_SOQ_PMON_BOX_STATUS, 
MSR_C7_PMON_BOX_STATUS, etc. 


e = Multi-counter overflow control MSRs: MSR_U_PMON_GLOBAL_OVF_CTL, MSR_SO_PMON_BOX_OVF_CTL, 
MSR_C7_PMON_BOX_OVF_CTL, etc. 

¢ Performance counters MSRs: the scope is implicitly per counter, e.g. MSR_U_PMON_CTR, 
MSR_SO_PMON_CTRO, MSR_C7_PMON_CTRS, etc. 


¢ Event select MSRs: the scope is implicitly per counter, e.g. MSR_U_PMON_EVNT_SEL, 
MSR_SO_PMON_EVNT_SELO, MSR_C7_PMON_EVNT_SEL5, etc 


¢ Sub-control MSRs: the scope is implicitly per-box granularity, e.g. MSR_MO_PMON_TIMESTAMP, 
MSR_RO_PMON_IPERFO_P1,MSR_S1_PMON_MATCH. 


Details of uncore PMU MSR bit field definitions can be found in a separate document “Intel Xeon Processor 7500 
Series Uncore Performance Monitoring Guide”. 


18.3.2 Performance Monitoring for Processors Based on Intel® Microarchitecture Code Name 
Westmere 


All of the performance monitoring programming interfaces (architectural and non-architectural core PMU facilities, 
and uncore PMU) described in Section 18.6.3 also apply to processors based on Intel® microarchitecture code 
name Westmere. 


Table 18-5 describes a non-architectural performance monitoring event (event code 0B7H) and associated 
MSR_OFFCORE_RSP_0O (address 1A6H) in the core PMU. This event and a second functionally equivalent offcore 
response event using event code OBBH and MSR_OFFCORE_RSP_1 (address 1A7H) are supported in processors 
based on Intel microarchitecture code name Westmere. The event code and event mask definitions of non-archi- 
tectural performance monitoring events can be found at: https://perfmon-events.intel.com/. 


The load latency facility is the same as described in Section 18.3.1.1.2, but added enhancement to provide more 
information in the data source encoding field of each load latency record. The additional information relates to 
STLB_MISS and LOCK, see Table 18-13. 


18.3.3. Intel® Xeon® Processor €7 Family Performance Monitoring Facility 


The performance monitoring facility in the processor core of the Intel® Xeon® processor E7 family is the same as 
those supported in the Intel Xeon processor 5600 series?. The uncore subsystem in the Intel Xeon processor E7 
family is similar to those of the Intel Xeon processor 7500 series. The high level construction of the uncore sub- 
system is similar to that shown in Figure 18-25, with the additional capability that up to 10 C-Box units are 
supported. 


3. Exceptions are indicated for event code OFH in the event list for this processor (https://perfmon-events.intel.com/); and valid 
bits of data source encoding field of each load latency record is limited to bits 5:4 of Table 18-13. 
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Table 18-9 summarizes the number MSRs for uncore PMU for each box. 


Table 18-9. Uncore PMU MSR Summary for Intel® Xeon® Processor E7 Family 


Counter General Global 

Box # of Boxes | Counters per Box Width Purpose Enable Sub-control MSRs 
C-Box 10 6 48 Yes per-box None 
S-Box 2 4 48 Yes per-box Match/Mask 
B-Box 2 4 48 Yes per-box Match/Mask 
M-Box 2 6 48 Yes per-box Yes 
R-Box 1 16 ( 2 port, 8 per port) 48 Yes per-box Yes 
W-Box 1 4 48 Yes per-box None 

1 48 No per-box None 
U-Box 1 1 48 Yes uncore None 


Details of the uncore performance monitoring facility of Intel Xeon Processor E7 family is available in the “Intel® 
Xeon® Processor E7 Uncore Performance Monitoring Programming Reference Manual”. 


18.3.4 Performance Monitoring for Processors Based on Intel® Microarchitecture Code Name 
Sandy Bridge 

Intel® Core™ i7-2xxx, Intel® Core™ i5-2xxx, Intel® Core™ i3-2xxx processor series, and Intel® Xeon® processor 
E3-1200 family are based on Intel microarchitecture code name Sandy Bridge; this section describes the perfor- 
mance monitoring facilities provided in the processor core. The core PMU supports architectural performance moni- 
toring capability with version ID 3 (see Section 18.2.3) and a host of non-architectural monitoring capabilities. 


Architectural performance monitoring version 3 capabilities are described in Section 18.2.3. 


The core PMU's capability is similar to those described in Section 18.3.1.1 and Section 18.6.3, with some differ- 
ences and enhancements relative to Intel microarchitecture code name Westmere summarized in Table 18-10. 


Table 18-10. Core PMU Comparison 
Intel® microarchitecture code 


Intel® microarchitecture code 


Box name Sandy Bridge name Westmere Comment 

# of Fixed counters per 3 3 Use CPUID to determine # of 
thread counters. See Section 18.2.1. 
# of general-purpose 8 8 Use CPUID to determine # of 
counters per core counters. See Section 18.2.1. 
Counter width (R,W) R:48, W: 32/48 R:48, W:32 See Section 18.2.2. 

# of programmable counters | 4 or (8 if acore not shared by two | 4 Use CPUID to determine # of 
per thread threads) counters. See Section 18.2.1. 
PMI Overhead Mitigation * Freeze_Perfmon_on_PMI with « Freeze_Perfmon_on_PMI See Section 17.4.7. 


legacy semantics. 

« Freeze_LBR_on_PMI with legacy 
semantics for branch profiling. 

* Freeze_while_SMM. 


with legacy semantics. 
Freeze_LBR_on_PMI with 
legacy semantics for branch 
profiling. 

* Freeze_while_SMM. 


See Table 18-78. 


Processor Event Based See Table 18-12. 


Sampling (PEBS) Events 


IA32_PMC4-IA32_PMC7 do 
not support PEBS. 
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Box 


Intel® microarchitecture code 
name Sandy Bridge 


Intel® microarchitecture code 
name Westmere 


Comment 


PEBS-Load Latency 


See Section 18.3.4.4.2; 


Data source encoding 


« Data source encoding 
« STLB miss encoding 
* Lock transaction encoding 


Section 18.3.4.4.3 No 


Yes (using precise No 
INST_RETIRED.ALL). 


MSR 1A6H and 1A7H, extended 
request and response types. 


PEBS-Precise Store 
PEBS-PDIR 


MSR 1A6H and 1A7H, limited 
response types. 


Nehalem supports 1A6H 
only. 


Off-core Response Event 


18.3.4.1 Global Counter Control Facilities In Intel® Microarchitecture Code Name Sandy Bridge 


The number of general-purpose performance counters visible to a logical processor can vary across Processors 
based on Intel microarchitecture code name Sandy Bridge. Software must use CPUID to determine the number 
performance counters/event select registers (See Section 18.2.1.1). 


ise LD 


63 35 34 33 3231 8 


| 
je 
7 ha 
ys 


FIXED_CTR2 enable 
FIXED_CTR1 enable 
FIXED_CTRO enable 
PMC7_EN (if PMC7 present) 
PMC6_EN (if PMC6 present) 
PMC5_EN (if PMC5 present) 
PMC4_EN (if PMC4 present) 
PMC3_EN 
PMC2_EN 
PMC1_EN 
PMCO_EN 


Reserved 8, else reserved. 


Valid if CPUID.OAH:EAX[15:8] = 


Figure 18-26. IA32_PERF_GLOBAL_CTRL MSR in Intel® Microarchitecture Code Name Sandy Bridge 


Figure 18-42 depicts the layout of IA32_PERF_GLOBAL_CTRL MSR. The enable bits (PMC4_EN, PMC5_EN, 
PMC6_EN, PMC7_EN) corresponding to IA32_PMC4-IA32_PMC7 are valid only if CPUID.OAH:EAX[15:8] reports a 
value of ‘8’. If CPUID.OAH:EAX[15:8] = 4, attempts to set the invalid bits will cause #GP. 


Each enable bit in IA32_PERF_GLOBAL_CTRL is AND’ed with the enable bits for all privilege levels in the respective 
IA32_PERFEVTSELx or IA32_PERF_FIXED_CTR_CTRL MSRs to start/stop the counting of respective counters. 
Counting is enabled if the AND’ed results is true; counting is disabled when the result is false. 
IA32_PERF_GLOBAL_STATUS MSR provides single-bit status used by software to query the overflow condition of 
each performance counter. IA32_PERF_GLOBAL_STATUS[bit 62] indicates overflow conditions of the DS area data 
buffer (see Figure 18-27). A value of 1 in each bit of the PMCx_OVF field indicates an overflow condition has 
occurred in the associated counter. 
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Figure 18-27. [A32_PERF_GLOBAL_STATUS MSR in Intel® Microarchitecture Code Name Sandy Bridge 


When a performance counter is configured for PEBS, an overflow condition in the counter will arm PEBS. On the 
subsequent event following overflow, the processor will generate a PEBS event. On a PEBS event, the processor will 
perform bounds checks based on the parameters defined in the DS Save Area (see Section 17.4.9). Upon 
successful bounds checks, the processor will store the data record in the defined buffer area, clear the counter 
overflow status, and reload the counter. If the bounds checks fail, the PEBS will be skipped entirely. In the event 
that the PEBS buffer fills up, the processor will set the OvfBuffer bit in MSR_PERF_GLOBAL_STATUS. 


IA32_PERF_GLOBAL_OVF_CTL MSR allows software to clear overflow the indicators for general-purpose or fixed- 


636261 


35 34 33 32 31 8 


CondChgd _| 
Ovf_DSBuffer 


Ovf_UncorePMU 


FIXED_CTR2 Overflow (RO) 
FIXED_CTR1 Overflow (RO) 


FIXED_CTRO Overflow (RO) 
PMC7_OVF (RO, If PMC7 present) 
PMC6_OVF (RO, If PMC6 present) 
PMC5_OVF (RO, If PMC5 present) 
PMC4_OVF (RO, If PMC4 present) 


PMC3_OVF (RO 


PMC2_OVF (RO 


) 
) 
PMC1_OVF (RO) 
) 


PMCO_OVF (RO 


Reserved 


Valid if CPUID.OAH:EAX[15:8] = 8; else reserved 


function counters via a single WRMSR (see Figure 18-28). Clear overflow indications when: 


° Setting up new values in the event select and/or UMASK field for counting or interrupt based sampling. 


* Reloading counter values to continue sampling. 
¢ Disabling event counting or interrupt based sampling. 


6362 


foe) 


35 34 33 32 31 


ClrCondChgd 

ClrOvfDSBuffer 
ClrOvfUncore — 
FIXED_CTR2 ClrOverflow 
FIXED_CTR1 ClrOverflow 


FIXED_CTRO ClrOverflow 


PMC7_ClrOvf (if PMC7 present) 
PMC6_ClrOvf (if PMC6 present) 


PMC5_ClrOvf (if PMC5 present) 


PMC4_ClrOvf (if PMC4 present) 


PMC3_ClrOvf 


PMC2_ClrOvf 


PMC1_ClrOvf 


PMCO_ClrOvf 


Reserved 


Valid if CPUID.OAH:EAX[15:8] = 8; else reserved 


Figure 18-28. IA32_PERF_GLOBAL_OVF_CTRL MSR in Intel microarchitecture code name Sandy Bridge 
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18.3.4.2 Counter Coalescence 


In processors based on Intel microarchitecture code name Sandy Bridge, each processor core implements eight 
general-purpose counters. CPUID.OAH:EAX[15:8] will report the number of counters visible to software. 


If a processor core is shared by two logical processors, each logical processors can access up to four counters 
(IA32_PMCO-IA32_PMC3). This is the same as in the prior generation for processors based on Intel microarchitec- 
ture code name Nehalem. 


If a processor core is not shared by two logical processors, up to eight general-purpose counters are visible. If 
CPUID.OAH:EAX[15:8] reports 8 counters, then IA32_PMC4-IA32_PMC7 would occupy MSR addresses OCSH 
through OC8H. Each counter is accompanied by an event select MSR (IA32_PERFEVTSEL4-IA32_PERFEVTSEL/7). 


If CPUID.OAH:EAX[15:8] report 4, access to IA32_PMC4-IA32_PMC7, IA32_PMC4-IA32_PMC7 will cause #GP. 
Writing 1’s to bit position 7:4 of IA32_PERF_GLOBAL_CTRL, IA32_PERF_GLOBAL_STATUS, or 
IA32_PERF_GLOBAL_OVF_CTL will also cause #GP. 


18.3.4.3 Full Width Writes to Performance Counters 


Processors based on Intel microarchitecture code name Sandy Bridge support full-width writes to the general- 
purpose counters, IA32_PMCx. Support of full-width writes are enumerated by 
IA32_PERF_CAPABILITIES.FW_WRITES[13] (see Section 18.2.4). 


The default behavior of IA32_PMCx is unchanged, i.e. WRMSR to IA32_PMCx results in a sign-extended 32-bit 
value of the input EAX written into IA32_PMCx. Full-width writes must issue WRMSR to a dedicated alias MSR 
address for each IA32_PMCx. 


Software must check the presence of full-width write capability and the presence of the alias address 
IA32_A_PMCx by testing IA32_PERF_CAPABILITIES[13]. 


18.3.4.4 PE€BS Support in Intel® Microarchitecture Code Name Sandy Bridge 


Processors based on Intel microarchitecture code name Sandy Bridge support PEBS, similar to those offered in 
prior generation, with several enhanced features. The key components and differences of PEBS facility relative to 
Intel microarchitecture code name Westmere is summarized in Table 18-11. 


Table 18-11. PEBS Facility Comparison 


Intel® microarchitecture code name __| Intel® microarchitecture 
Box Sandy Bridge code name Westmere Comment 
Valid IA32_PMCx PMCO-PMC3 PMCO-PMC3 No PEBS on PMC4-PMC7. 
PEBS Buffer Programming | Section 18.3.1.1.1 Section 18.3.1.1.1 Unchanged 
IA32_PEBS_ENABLE Figure 18-29 Figure 18-15 
Layout 
PEBS record layout Physical Layout same as Table 18-3. Table 18-3 Enhanced fields at offsets 98H, 
AOH, A8H. 
PEBS Events See Table 18-12. See Table 18-78. IA32_PMC4-IA32_PMC7 do not 
support PEBS. 
PEBS-Load Latency See Table 18-13. Table 18-4 
PEBS-Precise Store Yes; see Section 18.3.4.4.3. No IA32_PMC3 only 
PEBS-PDIR Yes No 1A32_PMC1 only 
PEBS skid from EventingIP | 1 (or 2 if micro+macro fusion) 1 
SAMPLING Restriction Small SAV(CountDown) value incur higher overhead than prior 
generation. 


Only IA32_PMCO through IA32_PMC3 support PEBS. 
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NOTE 


PEBS events are only valid when the following fields of IA32_PERFEVTSELx are all zero: AnyThread, 
Edge, Invert, CMask. 


In a PMU with PDIR capability, PEBS behavior is unpredictable if IA32_PERFEVTSELx or IA32_PMCx 
is changed for a PEBS-enabled counter while an event is being counted. To avoid this, changes to 
the programming or value of a PEBS-enabled counter should be performed when the counter is 
disabled. 


In IA32_PEBS_ENABLE MSR, bit 63 is defined as PS_ENABLE: When set, this enables IA32_PMC3 to capture 
precise store information. Only IA32_PMC3 supports the precise store facility. In typical usage of PEBS, the bit 
fields in IA32_PEBS_ENABLE are written to when the agent software starts PEBS operation; the enabled bit fields 
should be modified only when re-programming another PEBS event or cleared when the agent uses the perfor- 
mance counters for non-PEBS operations. 


63 62 36 3534 33 32 31 87654321 0 


PS_EN (R/W) 
LL_EN_PMC3 (RW) 
LL_EN_PMC2 (RW) 
LL_EN_PMC1 (RW) 
LL_EN_PMCO (R/W) 
PEBS_EN PMC3 (R/W 
PEBS EN PMC2 (R/W 
PEBS_EN PMC1 (R/W 
PEBS_EN PMCO (R/W 
Reserved RESET Value — 00000000_00000000H 


Figure 18-29. Layout of IA32_PEBS_ENABLE MSR 


18.3.4.4.1  PEBS Record Format 


The layout of PEBS records physically identical to those shown in Table 18-3, but the fields at offset 98H, AOH and 
A8H have been enhanced to support additional PEBS capabilities. 


Load/Store Data Linear Address (Offset 98H): This field will contain the linear address of the source of the load, 
or linear address of the destination of the store. 


Data Source /Store Status (Offset AOH): When load latency is enabled, this field will contain three piece of 
information (including an encoded value indicating the source which satisfied the load operation). The source 
field encodings are detailed in Table 18-4. When precise store is enabled, this field will contain information 
indicating the status of the store, as detailed in Table 19. 


Latency Value/0 (Offset A8H): When load latency is enabled, this field contains the latency in cycles to service 
the load. This field is not meaningful when precise store is enabled and will be written to zero in that case. Upon 
writing the PEBS record, microcode clears the overflow status bits in the IA32_PERF_GLOBAL_STATUS corre- 
sponding to those counters that both overflowed and were enabled in the IA32_PEBS_ENABLE register. The 
status bits of other counters remain unaffected. 


The number PEBS events has expanded. The list of PEBS events supported in Intel microarchitecture code name 
Sandy Bridge is shown in Table 18-12. 
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Table 18-12. PEBS Performance Events for Intel® Microarchitecture Code Name Sandy Bridge 


Event Name Event Select Sub-event UMask 
INST_RETIRED COH PREC_DIST 01H" 
UOPS_RETIRED C2H All 01H 
Retire_Slots 02H 
BR_INST_RETIRED C4H Conditional 01H 
Near_Call 02H 
All_branches 04H 
Near_Return 08H 
Near_Taken 20H 
BR_MISP_RETIRED C5H Conditional 01H 
Near_Call 02H 
All_branches 04H 
Not_Taken 10H 
Taken 20H 
MEM_UOPS_RETIRED DOH STLB_MISS_LOADS 11H 
STLB_MISS_STORE 12H 
LOCK_LOADS 21H 
SPLIT_LOADS 41H 
SPLIT_STORES 42H 
ALL_LOADS 81H 
ALL_STORES 82H 
MEM_LOAD_UOPS_RETIRED D1H L1_Hit 01H 
L2_Hit 02H 
L3_Hit 04H 
Hit_LFB 40H 
MEM_LOAD_UOPS_LLC_HIT_RETIRED D2H XSNP_Miss 01H 
XSNP_Hit 02H 
XSNP_Hitm 04H 
XSNP_None 08H 
NOTES: 


1. Only available on [A32_PMC1. 


18.3.4.4.2 Load Latency Performance Monitoring Facility 


The load latency facility in Intel microarchitecture code name Sandy Bridge is similar to that in prior microarchitec- 
ture. It provides software a means to characterize the average load latency to different levels of cache/memory 
hierarchy. This facility requires processor supporting enhanced PEBS record format in the PEBS buffer, see 

Table 18-3 and Section 18.3.4.4.1. This field measures the load latency from load's first dispatch of till final data 
writeback from the memory subsystem. The latency is reported for retired demand load operations and in core 
cycles (it accounts for re-dispatches). 


To use this feature software must assure: 


° One of the IA32_PERFEVTSELx MSR is programmed to specify the event unit MEM_TRANS_RETIRED, and the 
LATENCY_ABOVE_THRESHOLD event mask must be specified (IA32_PerfEvtSelX[15:0] = 1CDH). The corre- 
sponding counter IA32_PMCx will accumulate event counts for architecturally visible loads which exceed the 
programmed latency threshold specified separately in a MSR. Stores are ignored when this event is 
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programmed. The CMASK or INV fields of the IA32_PerfEvtSelX register used for counting load latency must be 
0. Writing other values will result in undefined behavior. 


¢ The MSR_PEBS_LD_LAT_THRESHOLD MSR is programmed with the desired latency threshold in core clock 
cycles. Loads with latencies greater than this value are eligible for counting and latency data reporting. The 
minimum value that may be programmed in this register is 3 (the minimum detectable load latency is 4 core 
clock cycles). 


¢ The PEBS enable bit in the IA32_PEBS_ENABLE register is set for the corresponding IA32_PMCx counter 
register. This means that both the PEBS_EN_CTRX and LL_EN_CTRX bits must be set for the counter(s) of 
interest. For example, to enable load latency on counter IA32_PMCO, the IA32_PEBS_ENABLE register must be 
programmed with the 64-bit value 00000001.00000001H. 


¢ When Load latency event is enabled, no other PEBS event can be configured with other counters. 


When the load-latency facility is enabled, load operations are randomly selected by hardware and tagged to carry 
information related to data source locality and latency. Latency and data source information of tagged loads are 
updated internally. The MEM_TRANS_RETIRED event for load latency counts only tagged retired loads. If a load is 
cancelled it will not be counted and the internal state of the load latency facility will not be updated. In this case the 
hardware will tag the next available load. 


When a PEBS assist occurs, the last update of latency and data source information are captured by the assist and 
written as part of the PEBS record. The PEBS sample after value (SAV), specified in PEBS CounterX Reset, operates 
orthogonally to the tagging mechanism. Loads are randomly tagged to collect latency data. The SAV controls the 
number of tagged loads with latency information that will be written into the PEBS record field by the PEBS assists. 
The load latency data written to the PEBS record will be for the last tagged load operation which retired just before 
the PEBS assist was invoked. 


The physical layout of the PEBS records is the same as shown in Table 18-3. The specificity of Data Source entry at 
offset AOH has been enhanced to report three pieces of information. 


Table 18-13. Layout of Data Source Field of Load Latency Record 


Field Position Description 

Source 3:0 See Table 18-4 

STLB_MISS 4 0: The load did not miss the STLB (hit the DTLB or STLB). 
1: The load missed the STLB. 

Lock 5 0: The load was not part of a locked transaction. 
1: The load was part of a locked transaction. 

Reserved 63:6 Reserved 


The layout of MSR_PEBS_LD_LAT_THRESHOLD is the same as shown in Figure 18-17. 


18.3.4.4.3 Precise Store Facility 


Processors based on Intel microarchitecture code name Sandy Bridge offer a precise store capability that comple- 
ments the load latency facility. It provides a means to profile store memory references in the system. 


Precise stores leverage the PEBS facility and provide additional information about sampled stores. Having precise 
memory reference events with linear address information for both loads and stores can help programmers improve 
data structure layout, eliminate remote node references, and identify cache-line conflicts in NUMA systems. 


Only IA32_PMC3 can be used to capture precise store information. After enabling this facility, counter overflows will 
initiate the generation of PEBS records as previously described in PEBS. Upon counter overflow hardware captures 
the linear address and other status information of the next store that retires. This information is then written to the 
PEBS record. 


To enable the precise store facility, software must complete the following steps. Please note that the precise store 
facility relies on the PEBS facility, so the PEBS configuration requirements must be completed before attempting to 
capture precise store information. 


* Complete the PEBS configuration steps. 
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¢ Program the MEM_TRANS_RETIRED.PRECISE_STORE event in IA32_PERFEVTSEL3. Only counter 3 
(IA32_PMC3) supports collection of precise store information. 


° =Set IA32_PEBS_ENABLE[3] and IA32_PEBS_ENABLE[63]. This enables IA32_PMC3 as a PEBS counter and 
enables the precise store facility, respectively. 


The precise store information written into a PEBS record affects entries at offset 98H, AOH and A8H of Table 18-3. 
The specificity of Data Source entry at offset AOH has been enhanced to report three piece of information. 


Table 18-14. Layout of Precise Store Information In PEBS Record 
Field Offset | Description 


Store Data 98H The linear address of the destination of the store. 
Linear Address 


Store Status AOH L1D Hit (Bit 0): The store hit the data cache closest to the core (lowest latency cache) if this bit is set, 
otherwise the store missed the data cache. 
STLB Miss (bit 4): The store missed the STLB if set, otherwise the store hit the STLB 


Locked Access (bit 5): The store was part of a locked access if set, otherwise the store was not part of a 
locked access. 


Reserved A8H Reserved 


18.3.4.4.4 Precise Distribution of Instructions Retired (PDIR) 


Upon triggering a PEBS assist, there will be a finite delay between the time the counter overflows and when the 
microcode starts to carry out its data collection obligations. INST_RETIRED is a very common event that is used to 
sample where performance bottleneck happened and to help identify its location in instruction address space. Even 
if the delay is constant in core clock space, it invariably manifest as variable “skids” in instruction address space. 
This creates a challenge for programmers to profile a workload and pinpoint the location of bottlenecks. 


The core PMU in processors based on Intel microarchitecture code name Sandy Bridge include a facility referred to 
as precise distribution of Instruction Retired (PDIR). 


The PDIR facility mitigates the “skid” problem by providing an early indication of when the INST_RETIRED counter 
is about to overflow, allowing the machine to more precisely trap on the instruction that actually caused the 
counter overflow. On processors based on Intel microarchitecture code name Sandy Bridge skid is significantly 
reduced, and can be as little as one instruction. On future implementations PDIR may eliminate skid. 


PDIR applies only to the INST_RETIRED.ALL precise event, and processors based on Sandy Bridge microarchitec- 
ture must use IA32_PMC1 with PerfEvtSel1 property configured and bit 1 in the IA32_PEBS_ENABLE set to 1. 
INST_RETIRED.ALL is a non-architectural performance event, it is not supported in prior generation microarchitec- 
tures. Additionally, on processors with CPUID DisplayFamily_DisplayModel signatures of 06_2A and 06_2D, the tool 
that programs PDIR should quiesce the rest of the programmable counters in the core when PDIR is active. 


18.3.4.5 Off-core Response Performance Monitoring 


The core PMU in processors based on Intel microarchitecture code name Sandy Bridge provides off-core response 
facility similar to prior generation. Off-core response can be programmed only with a specific pair of event select 
and counter MSR, and with specific event codes and predefine mask bit value in a dedicated MSR to specify attri- 
butes of the off-core transaction. Two event codes are dedicated for off-core response event programming. Each 
event code for off-core response monitoring requires programming an associated configuration MSR, 
MSR_OFFCORE_RSP_x. Table 18-15 lists the event code, mask value and additional off-core configuration MSR 
that must be programmed to count off-core response events using IA32_PMCx. 
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Table 18-15. Off-Core Response Event Encoding 


Counter Event code UMask Required Off-core Response MSR 
PMCO-3 B7H 01H MSR_OFFCORE_RSP_O (address 1A6H) 
PMCO-3 BBH 01H MSR_OFFCORE_RSP_1 (address 1A7H) 


The layout of MSR_| 


OFFCORE_RSP_0O and MSR_OFFCORE_RSP_1i are shown in Figure 18-30 and Figure 18-31. Bits 


15:0 specifies the request type of a transaction request to the uncore. Bits 30:16 specifies supplier information, 
bits 37:31 specifies snoop response information. 


RESPONSE TYPE — Other (R/W) | 


RESERVED 
REQUEST TYPE — STRM_ST (RW) 
REQUEST TYPE — BUS_LOCKS (R/W) 
REQUEST TYPE — PF_LLC_IFETCH (R/W) 
REQUEST TYPE — PF_LLC_RFO (R/W) 
REQUEST TYPE — PF_LLC_DATA RD (R/W) 
REQUEST TYPE — PF_IFETCH (R/W) 
REQUEST TYPE — PF_RFO (R/W) 
REQUEST TYPE — PF_DATA_RD (R/W) 
REQUEST TYPE — WB (R/W) 
REQUEST TYPE — DMND_IFETCH (R/W) 
REQUEST TYPE — DMND_RFO (R/W) 
REQUEST TYPE — DMND_DATA_RD (R/W) 


63 37 15 1413121110 9876 54321 0 


See Figure 18-30 


Reserved 


RESET Value — 00000000_00000000H 


Figure 18-30. Request_Type Fields for MSR_OFFCORE_RSP_x 


Table 18-16. MSR_OFFCORE_RSP_x Request_Type Field Definition 


Bit Name Offset | Description 

DMND_DATA_RD | 0 Counts the number of demand data reads of full and partial cachelines as well as demand data page 
table entry cacheline reads. Does not count L2 data read prefetches or instruction fetches. 

DMND_RFO 1 Counts the number of demand and DCU prefetch reads for ownership (RFO) requests generated by a 
write to data cacheline. Does not count L2 RFO prefetches. 

DMND_IFETCH 2 Counts the number of demand instruction cacheline reads and L1 instruction cacheline prefetches. 

WB 3 Counts the number of writeback (modified to exclusive) transactions. 

PF_DATA_RD 4 Counts the number of data cacheline reads generated by L2 prefetchers. 

PF_RFO 5 Counts the number of RFO requests generated by L2 prefetchers. 

PF_IFETCH 6 Counts the number of code reads generated by L2 prefetchers. 

PF_LLC_DATA_RD | 7 L2 prefetcher to L3 for loads. 

PF_LLC_RFO 8 RFO requests generated by L2 prefetcher 

PF_LLC_IFETCH 9 L2 prefetcher to L3 for instruction fetches. 

BUS_LOCKS 10 Bus lock and split lock requests 

STRM_ST 11 Streaming store requests 

OTHER 15 Any other request that crosses IDI, including I/O. 
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63 


37 36 35 34 33 32 31 


22 212019 18 17 16 


RESPONSE TYPE — NON_DRAM (R/W) ——— 
RSPNS_SNOOP — HITM (RW) = ——— 
RSPNS_SNOOP — HIT_FWD 
RSPNS_SNOOP — HIT_NO_FWD (RW) 
RSPNS_SNOOP — SNP_MISS (R/W) 
RSPNS_SNOOP — SNP_NOT_NEEDED (R/W) 
RSPNS_SNOOP — SNPI_NONE (R/W) 
RSPNS_SUPPLIER — RESERVED 
RSPNS_ SUPPLIER — Local 
RSPNS_SUPPLIER — LLC_HITF (R/W) 
RSPNS_ SUPPLIER — LLC_HITS (R/W) 
RSPNS_ SUPPLIER — LLC_HITE (R/W) 
RSPNS_SUPPLIER — LLC_HITM (R/W) 
RSPNS_SUPPLIER — No_ SUPP (R/W) 
RSPNS_ SUPPLIER — ANY (R/W) 


Reserved 


RESET Value — 00000000_00000000H 


Figure 18-31. Response_Supplier and Snoop Info Fields for MSR_OFFCORE_RSP_x 


To properly program this extra register, software must set at least one request type bit and a valid response type 
pattern. Otherwise, the event count reported will be zero. It is permissible and useful to set multiple request and 
response type bits in order to obtain various classes of off-core response events. Although MSR_OFFCORE_RSP_x 
allow an agent software to program numerous combinations that meet the above guideline, not all combinations 
produce meaningful data. 


Table 18-17. MSR_OFFCORE_RSP_x Response Supplier Info Field Definition 


Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier NO_SUPP 17 No Supplier Information available. 
Info LLC_HITM 18 M-state initial lookup stat in L3. 
LLC_HITE 19 E-state 
LLC_HITS 20 S-state 
LLC_HITF 21 F-state 
LOCAL 22 Local DRAM Controller. 
Reserved 30:23 Reserved 


To specify a complete offcore response filter, software must properly program bits in the request and response type 
fields. A valid request type must have at least one bit set in the non-reserved bits of 15:0. A valid response type 
must be a non-zero value of the following expression: 


ANY | [COR’ of Supplier Info Bits) & ((OR’ of Snoop Info Bits) ] 
If “ANY™ bit is set, the supplier and snoop info bits are ignored. 
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Table 18-18. MSR_OFFCORE_RSP_x Snoop Info Field Definition 


Subtype | Bit Name Offset | Description 
Snoop SNP_NONE 31 No details on snoop-related information. 
Info 
SNP_NOT_NEEDED | 32 No snoop was needed to satisfy the request. 
SNP_MISS 33 A snoop was needed and it missed all snooped caches: 
-For LLC Hit, ReslHitl was returned by all cores 
-For LLC Miss, Rspl was returned by all sockets and data was returned from DRAM. 
SNP_NO_FWD 34 A snoop was needed and it hits in at least one snooped cache. Hit denotes a cache-line was 
valid before snoop effect. This includes: 
-Snoop Hit w/ Invalidation (LLC Hit, RFO) 
-Snoop Hit, Left Shared (LLC Hit/Miss, IFetch/Data_RD) 
-Snoop Hit w/ Invalidation and No Forward (LLC Miss, RFO Hit S) 
In the LLC Miss case, data is returned from DRAM. 
SNP_FWD 35 A snoop was needed and data was forwarded from a remote socket. This includes: 
-Snoop Forward Clean, Left Shared (LLC Hit/Miss, IFetch/Data_RD/RFT). 
HITM 36 A snoop was needed and it HitM-ed in local or remote cache. HitM denotes a cache-line was 
in modified state before effect as a results of snoop. This includes: 
-Snoop HitM w/ WB (LLC miss, IFetch/Data_RD) 
-Snoop Forward Modified w/ Invalidation (LLC Hit/Miss, RFO) 
-Snoop MtoS (LLC Hit, IFetch/Data_RD). 
NON_DRAM 37 Target was non-DRAM system address. This includes MMIO transactions. 


18.3.4.6 Uncore Performance Monitoring Facilities In Intel® Core’ i7-2xxx, Intel® Core™ i5-2xxx, 
Intel® Core™” i3-2xxx Processor Series 


The uncore sub-system in Intel® Core™ i7-2xxx, Intel® Core™ i5-2xxx, Intel® Core™ i3-2xxx processor series 
provides a unified L3 that can support up to four processor cores. The L3 cache consists multiple slices, each slice 
interface with a processor via a coherence engine, referred to as a C-Box. Each C-Box provides dedicated facility of 
MSRs to select uncore performance monitoring events and each C-Box event select MSR is paired with a counter 

register, similar in style as those described in Section 18.3.1.2.2. The ARB unit in the uncore also provides its local 
performance counters and event select MSRs. The layout of the event select MSRs in the C-Boxes and the ARB unit 
are shown in Figure 18-32. 


63 28 24 23 22 21 201918171615 87 0 


Guha Unit Mask (UMASK)| _ Event Select 


INV—lInvert counter mask—_ 
EN—Enable counter —————_- 
OVF_EN—Overflow forwarding 


E—Edge detect 
Reserved 


RESET Value — 00000000_00000000H 


Figure 18-32. Layout of Uncore PERFEVTSEL MSR for a C-Box Unit or the ARB Unit 
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The bit fields of the uncore event select MSRs for a C-box unit or the ARB unit are summarized below: 


Event_Select (bits 7:0) and UMASK (bits 15:8): Specifies the microarchitectural condition to count in a local 
uncore PMU counter, see the event list at: https://perfmon-events.intel.com/. 
E (bit 18): Enables edge detection filtering, if 1. 


OVF_EN (bit 20): Enables the overflow indicator from the uncore counter forwarded to 
MSR_UNC_PERF_GLOBAL_CTRL, if 1. 


EN (bit 22): Enables the local counter associated with this event select MSR. 
INV (bit 23): Event count increments with non-negative value if 0, with negated value if 1. 
CMASK (bits 28:24): Specifies a positive threshold value to filter raw event count input. 


At the uncore domain level, there is a master set of control MSRs that centrally manages all the performance moni- 
toring facility of uncore units. Figure 18-33 shows the layout of the uncore domain global control. 


When an uncore counter overflows, a PMI can be routed to a processor core. Bits 3:0 of 
MSR_UNC_PERF_GLOBAL_CTRL can be used to select which processor core to handle the uncore PMI. Software 
must then write to bit 13 of IA32_DEBUGCTL (at address 1D9H) to enable this capability. 


PMI_SEL_Core#: Enables the forwarding of an uncore PMI request to a processor core, if 1. If bit 30 (WakePMI) 
is ‘1’, a wake request is sent to the respective processor core prior to sending the PMI. 


EN: Enables the fixed uncore counter, the ARB counters, and the CBO counters in the uncore PMU, if 1. This bit 
is cleared if bit 31 (FREEZE) is set and any enabled uncore counters overflow. 


WakePMI: Controls sending a wake request to any halted processor core before issuing the uncore PMI request. 
If a processor core was halted and not sent a wake request, the uncore PMI will not be serviced by the 
processor core. 


FREEZE: Provides the capability to freeze all uncore counters when an overflow condition occurs in a unit 
counter. When this bit is set, and a counter overflow occurs, the uncore PMU logic will clear the global enable 
bit (bit 29). 


63 32 31 30 29 28 4321 0 


FREEZE—Freeze counters 
WakePMI—Wake cores on PMI 
EN—Enable all uncore counters 
PMI_Sel_Core3 — Uncore PMI to core 3 
PMI_Sel_Core2 — Uncore PMI to core 2 


PMI_Sel_Core1 — Uncore PMI to core 1 
PMI_Sel_CoreO — Uncore PMI to core 0 


Reserved RESET Value — 00000000_00000000H 


Figure 18-33. Layout of MSR_UNC_PERF_GLOBAL_CTRL MSR for Uncore 
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Additionally, there is also a fixed counter, counting uncore clockticks, for the uncore domain. Table 18-19 summa- 
rizes the number MSRs for uncore PMU for each box. 


Table 18-19. Uncore PMU MSR Summary 


Counters per | Counter General Global 
Box # of Boxes Box Width Purpose Enable Comment 
C-Box SKU specific 2 44 Yes Per-box Up to 4, seeTable 2-21 
MSR_UNC_CBO_CONFIG 
ARB 1 2 44 Yes Uncore 
Fixed NA. N.A. 48 No Uncore 
Counter 


18.3.4.6.1 Uncore Performance Monitoring Events 

There are certain restrictions on the uncore performance counters in each C-Box. Specifically, 
® Occupancy events are supported only with counter O but not counter 1. 

¢ Other uncore C-Box events can be programmed with either counter 0 or 1. 


The C-Box uncore performance events can collect performance characteristics of transactions initiated by 
processor core. In that respect, they are similar to various sub-events in the OFFCORE_RESPONSE family of perfor- 
mance events in the core PMU. Information such as data supplier locality (LLC HIT/MISS) and snoop responses can 
be collected via OFFCORE_RESPONSE and qualified on a per-thread basis. 


On the other hand, uncore performance event logic cannot associate its counts with the same level of per-thread 
qualification attributes as the core PMU events can. Therefore, whenever similar event programming capabilities 

are available from both core PMU and uncore PMU, the recommendation is that utilizing the core PMU events may 
be less affected by artifacts, complex interactions and other factors. 


18.3.4.7 Intel® Xeon® Processor €5 Family Performance Monitoring Facility 


The Intel® Xeon® Processor E5 Family (and Intel® Core™ i7-3930K Processor) are based on Intel microarchitec- 
ture code name Sandy Bridge-E. While the processor cores share the same microarchitecture as those of the Intel® 
Xeon® Processor E3 Family and 2nd generation Intel Core i7-2xxx, Intel Core i5-2xxx, Intel Core i3-2xxx processor 
series, the uncore subsystems are different. An overview of the uncore performance monitoring facilities of the 
Intel Xeon processor E5 family (and Intel Core i7-3930K processor) is described in Section 18.3.4.8. 


Thus, the performance monitoring facilities in the processor core generally are the same as those described in 
Section 18.6.3 through Section 18.3.4.5. However, the MSR_OFFCORE_RSP_0/MSR_OFFCORE_RSP_1 Response 
Supplier Info field shown in Table 18-17 applies to Intel Core Processors with CPUID signature of 
DisplayFamily_DisplayModel encoding of 06_2AH; Intel Xeon processor with CPUID signature of 
DisplayFamily_DisplayModel encoding of 06_2DH supports an additional field for remote DRAM controller shown in 
Table 18-20. Additionally, there are some small differences in the non-architectural performance monitoring events 
(see event list available at: https://perfmon-events.intel.com/). 
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Table 18-20. MSR_OFFCORE_RSP_x Supplier Info Field Definitions 


Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier Info | NO_SUPP 17 No Supplier Information available. 
LLC_HITM 18 M-state initial lookup stat in L3. 
LLC_HITE 19 €-state 
LLC_HITS 20 S-state 
LLC_HITF 21 F-state 
LOCAL 22 Local DRAM Controller. 
Remote 30:23 Remote DRAM Controller (either all Os or all 1s). 


18.3.4.8 Intel® Xeon® Processor €5 Family Uncore Performance Monitoring Facility 


The uncore subsystem in the Intel Xeon processor E5-2600 product family has some similarities with those of the 
Intel Xeon processor E7 family. Within the uncore subsystem, localized performance counter sets are provided at 
logic control unit scope. For example, each Cbox caching agent has a set of local performance counters, and the 
power controller unit (PCU) has its own local performance counters. Up to 8 C-Box units are supported in the 
uncore sub-system. 


Table 18-21 summarizes the uncore PMU facilities providing MSR interfaces. 


Table 18-21. Uncore PMU MSR Summary for Intel® Xeon® Processor E5 Family 


Counter General Global 
Box # of Boxes | Counters per Box Width Purpose Enable Sub-control MSRs 
C-Box 8 4 44 Yes per-box None 
PCU 1 4 48 Yes per-box Match/Mask 
U-Box 1 2 44 Yes uncore None 


Details of the uncore performance monitoring facility of Intel Xeon Processor E5 family is available in “Intel® 
Xeon® Processor E5 Uncore Performance Monitoring Programming Reference Manual”. The MSR-based uncore 
PMU interfaces are listed in Table 2-24. 


18.3.5 3rd Generation Intel® Core’ Processor Performance Monitoring Facility 


The 3rd generation Intel® Core™ processor family and Intel® Xeon® processor E3-1200v2 product family are 
based on the Ivy Bridge microarchitecture. The performance monitoring facilities in the processor core generally 
are the same as those described in Section 18.6.3 through Section 18.3.4.5. The non-architectural performance 
monitoring events supported by the processor core can be found at: https://perfmon-events.intel.com/. 


18.3.5.1 Intel® Xeon® Processor E5 v2 and €7 v2 Family Uncore Performance Monitoring Facility 


The uncore subsystem in the Intel Xeon processor E5 v2 and Intel Xeon Processor E7 v2 product families are based 
on the Ivy Bridge-E microarchitecture. There are some similarities with those of the Intel Xeon processor E5 family 
based on the Sandy Bridge microarchitecture. Within the uncore subsystem, localized performance counter sets 
are provided at logic control unit scope. 


Details of the uncore performance monitoring facility of Intel Xeon Processor E5 v2 and Intel Xeon Processor E7 v2 
families are available in “Intel® Xeon® Processor E5 v2 and E7 v2 Uncore Performance Monitoring Programming 
Reference Manual”. The MSR-based uncore PMU interfaces are listed in Table 2-28. 


Vol. 3B 18-47 


PERFORMANCE MONITORING 


18.3.6 


4th Generation Intel® Core™ Processor Performance Monitoring Facility 


The 4th generation Intel® Core™ processor and Intel® Xeon® processor E3-1200 v3 product family are based on 
the Haswell microarchitecture. The core PMU supports architectural performance monitoring capability with version 
ID 3 (see Section 18.2.3) and a host of non-architectural monitoring capabilities. 


Architectural performance monitoring version 3 capabilities are described in Section 18.2.3. 


The core PMU’'s capability is similar to those described in Section 18.6.3 through Section 18.3.4.5, with some differ- 
ences and enhancements summarized in Table 18-22. Additionally, the core PMU provides some enhancement to 
support performance monitoring when the target workload contains instruction streams using Intel® Transactional 
Synchronization Extensions (TSX), see Section 18.3.6.5. For details of Intel TSX, see Chapter 16, “Programming with 
Intel® Transactional Synchronization Extensions” of Inte/® 64 and IA-32 Architectures Software Developer’s 


Manual, Volume 1. 


Table 18-22. Core PMU Comparison 


Intel® microarchitecture code 


Intel® microarchitecture code 


# of programmable counters per 
thread 


4 or (8 if a core not shared by two 
threads) 


4 or (8 if a core not shared by 
two threads) 


Box name Haswell name Sandy Bridge Comment 

# of Fixed counters per thread 3 3 Use CPUID to determine 
# of counters. See 
Section 18.2.1. 

# of general-purpose counters 8 8 Use CPUID to determine 

per core # of counters. See 
Section 18.2.1. 

Counter width (R,W) R:48, W: 32/48 R:48, W: 32/48 See Section 18.2.2. 


Use CPUID to determine 
# of counters. See 
Section 18.2.1. 


PMI Overhead Mitigation 


* Freeze_Perfmon_on_PMI with 
legacy semantics. 

« Freeze_LBR_on_PMI with legacy 
semantics for branch profiling. 

* Freeze_while_SMM. 


« Freeze_Perfmon_on_PMI 
with legacy semantics. 

* Freeze_LBR_on_PMI with 
legacy semantics for branch 
profiling. 

* Freeze_while_SMM. 


See Section 17.4.7. 


Processor Event Based Sampling 
(PEBS) Events 


See Table 18-12 and Section 
18.3.6.5.1. 


See Table 18-12. 


IA32_PMC4-IA32_PMC7 
do not support PEBS. 


PEBS-Load Latency 


See Section 18.3.4.4.2. 


See Section 18.3.4.4.2. 


PEBS-Precise Store 


No, replaced by Data Address 
profiling. 


Section 18.3.4.4.3 


PEBS-PDIR Yes (using precise Yes (using precise 
INST_RETIRED.ALL) INST_RETIRED.ALL) 
PEBS-EventingIP Yes No 
Data Address Profiling Yes No 
LBR Profiling Yes Yes 
Call Stack Profiling Yes, see Section 17.11. No Use LBR facility. 


Off-core Response Event 


MSR 1A6H and 1A7H; extended 
request and response types. 


MSR 1A6H and 1A7H; extended 
request and response types. 


Intel TSX support for Perfmon 


See Section 18.3.6.5. 


No 
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The PEBS facility in the 4th Generation Intel Core processor is similar to those in processors based on Intel micro- 
architecture code name Sandy Bridge, with several enhanced features. The key components and differences of 
PEBS facility relative to Intel microarchitecture code name Sandy Bridge is summarized in Table 18-23. 


Table 18-23. PEBS Facility Comparison 


Intel® microarchitecture code 


Intel® microarchitecture code 


Box name Haswell name Sandy Bridge Comment 

Valid IA32_PMCx PMCO-PMC3 PMCO-PMC3 No PEBS on PMC4-PMC7 
PEBS Buffer Programming Section 18.3.1.1.1 Section 18.3.1.1.1 Unchanged 
IA32_PEBS_ENABLE Layout | Figure 18-15 Figure 18-29 


PEBS record layout 


Table 18-24; enhanced fields at 
offsets 98H, AOH, A8H, BOH. 


Table 18-3; enhanced fields at 
offsets 98H, AOH, A8H. 


Precise Events 


PEBS-Load Latency 


See Table 18-12. 


See Table 18-13. 


See Table 18-12. 


Table 18-13 


IA32_PMC4-IA32_PMC7 do not 
support PEBS. 


PEBS-Precise Store 


No, replaced by data address 
profiling. 


Yes; see Section 18.3.4.4.3. 


PEBS-PDIR 


Yes 


Yes 


IA32_PMC1 only. 


PEBS skid from EventingIP 


1 (or 2 if micro+macro fusion) 


1 


SAMPLING Restriction 


Small SAV(CountDown) value incur higher overhead than prior 


generation. 


Only IA32_PMCO through IA32_PMC3 support PEBS. 


NOTE 


PEBS events are only valid when the following fields of IA32_PERFEVTSELx are all zero: AnyThread, 
Edge, Invert, CMask. 


In a PMU with PDIR capability, PEBS behavior is unpredictable if IA32_PERFEVTSELx or IA32_PMCx 
is changed for a PEBS-enabled counter while an event is being counted. To avoid this, changes to 
the programming or value of a PEBS-enabled counter should be performed when the counter is 


disabled. 


18.3.6.2 PEBS Data Format 


The PEBS record format for the 4th Generation Intel Core processor is shown in Table 18-24. The PEBS record 

format, along with debug/store area storage format, does not change regardless of whether IA-32e mode is active 
or not. CPUID.01H:ECX.DTES64[bit 2] reports whether the processor's DS storage format support is mode-inde- 
pendent. When set, it uses 64-bit DS storage format. 
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Table 18-24. PEBS Record Format for 4th Generation Intel Core Processor Family 


Byte Offset Field Byte Offset Field 

00H R/EFLAGS 60H R10 

08H R/EIP 68H R11 

10H R/EAX 70H R12 

18H R/EBX 78H R13 

20H R/ECX 80H R14 

28H R/EDX 88H R15 

30H R/ESI 90H IA32_PERF_GLOBAL_STATUS 

38H R/EDI 98H Data Linear Address 

40H R/EBP AOH Data Source Encoding 

48H R/ESP A8H Latency value (core cycles) 

50H R8 BOH EventingIP 

58H RQ B8H TX Abort Information (Section 
18.3.6.5.1) 


The layout of PEBS records are almost identical to those shown in Table 18-3. Offset BOH is a new field that records 
the eventing IP address of the retired instruction that triggered the PEBS assist. 


The PEBS records at offsets 98H, AOH, and ABH record data gathered from three of the PEBS capabilities in prior 
processor generations: load latency facility (Section 18.3.4.4.2), PDIR (Section 18.3.4.4.4), and the equivalent 
capability of precise store in prior generation (see Section 18.3.6.3). 


In the core PMU of the 4th generation Intel Core processor, load latency facility and PDIR capabilities are 
unchanged. However, precise store is replaced by an enhanced capability, data address profiling, that is not 
restricted to store address. Data address profiling also records information in PEBS records at offsets 98H, AOH, 
and ABH. 


18.3.6.3. PE€BS Data Address Profiling 


The Data Linear Address facility is also abbreviated as DataLA. The facility is a replacement or extension of the 
precise store facility in previous processor generations. The DataLA facility complements the load latency facility by 
providing a means to profile load and store memory references in the system, leverages the PEBS facility, and 
provides additional information about sampled loads and stores. Having precise memory reference events with 
linear address information for both loads and stores provides information to improve data structure layout, elimi- 
nate remote node references, and identify cache-line conflicts in NUMA systems. 


The DataLA facility in the 4th generation processor supports the following events configured to use PEBS: 


Table 18-25. Precise Events That Supports Data Linear Address Profiling 


Event Name Event Name 
MEM_UOPS_RETIRED.STLB_MISS_LOADS MEM_UOPS_RETIRED.STLB_MISS_STORES 
MEM_UOPS_RETIRED.LOCK_LOADS MEM_UOPS_RETIRED.SPLIT_STORES 
MEM_UOPS_RETIRED.SPLIT_LOADS MEM_UOPS_RETIRED.ALL_STORES 
MEM_UOPS_RETIRED.ALL_LOADS MEM_LOAD_UOPS_LLC_MISS_RETIRED.LOCAL_DRAM 
MEM_LOAD_UOPS_RETIRED.L1_HIT MEM_LOAD_UOPS_RETIRED.L2_HIT 
MEM_LOAD_UOPS_RETIRED.L3_HIT MEM_LOAD_UOPS_RETIRED.L1_MISS 
MEM_LOAD_UOPS_RETIRED.L2_MISS MEM_LOAD_UOPS_RETIRED.L3_MISS 
MEM_LOAD_UOPS_RETIRED.HIT_LFB MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_MISS 
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Table 18-25. Precise Events That Supports Data Linear Address Profiling (Contd.) 


Event Name Event Name 
MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_HIT MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_HITM 
UOPS_RETIRED.ALL (if load or store is tagged) MEM_LOAD_UOPS_LLC_HIT_RETIRED.XSNP_NONE 


DataLA can use any one of the IA32_PMCO0-IA32_PMC3 counters. Counter overflows will initiate the generation of 
PEBS records. Upon counter overflow, hardware captures the linear address and possible other status information 
of the retiring memory uop. This information is then written to the PEBS record that is subsequently generated. 


To enable the DataLA facility, software must complete the following steps. Please note that the DataLA facility relies 

on the PEBS facility, so the PEBS configuration requirements must be completed before attempting to capture 

DataLA information. 

* Complete the PEBS configuration steps. 

¢ Program an event listed in Table 18-25 using any one of IA32_PERFEVTSELO-IA32_PERFEVTSEL3. 

° Set the corresponding IA32_PEBS_ENABLE.PEBS_EN_CTRx bit. This enables the corresponding IA32_PMCx as 
a PEBS counter and enables the DataLA facility. 

When the DataLA facility is enabled, the relevant information written into a PEBS record affects entries at offsets 

98H, AOH and A&8H, as shown in Table 18-26. 


Table 18-26. Layout of Data Linear Address Information In PEBS Record 


Field Offset | Description 


Data Linear 98H The linear address of the load or the destination of the store. 
Address 


Store Status AOH 


DCU Hit (Bit 0): The store hit the data cache closest to the core (L1 cache) if this bit is set, otherwise 
the store missed the data cache. This information is valid only for the following store events: 
UOPS_RETIRED.ALL (if store is tagged), 

MEM_UOPS_RETIRED.STLB_MISS_STORES, 

MEM_UOPS_RETIRED.SPLIT_STORES, MEM_UOPS_RETIRED.ALL_STORES 

* Other bits are zero, The STLB_MISS, LOCK bit information can be obtained by programming the 
corresponding store event in Table 18-25. 


Reserved A8H Always zero. 


18.3.6.3.1 EventingIP Record 


The PEBS record layout for processors based on Intel microarchitecture code name Haswell adds a new field at 
offset OBOH. This is the eventingIP field that records the IP address of the retired instruction that triggered the 
PEBS assist. The EIP/RIP field at offset O8H records the IP address of the next instruction to be executed following 


the PEBS assist. 


18.3.6.4 Off-core Response Performance Monitoring 


The core PMU facility to collect off-core response events are similar to those described in Section 18.3.4.5. The 
event codes are listed in Table 18-15. Each event code for off-core response monitoring requires programming an 
associated configuration MSR, MSR_OFFCORE_RSP_x. Software must program MSR_OFFCORE_RSP_x according 


to: 

° Transaction request type encoding (bits 15:0): see Table 18-27. 
°® Supplier information (bits 30:16): see Table 18-28. 

¢ Snoop response information (bits 37:31): see Table 18-18. 
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Table 18-27. MSR_OFFCORE_RSP_x Request_Type Definition (Haswell microarchitecture) 

Bit Name Offset | Description 

DMND_DATA_RD | 0 Counts the number of demand data reads and page table entry cacheline reads. Does not count L2 data 
read prefetches or instruction fetches. 

DMND_RFO 1 Counts demand read (RFO) and software prefetches (PREFETCHW) for exclusive ownership in 
anticipation of a write. 

DMND_IFETCH 2 Counts the number of demand instruction cacheline reads and L1 instruction cacheline prefetches. 

COREWB 3 Counts the number of modified cachelines written back. 

PF_DATA_RD 4 Counts the number of data cacheline reads generated by L2 prefetchers. 

PF_RFO 5 Counts the number of RFO requests generated by L2 prefetchers. 

PF_IFETCH 6 Counts the number of code reads generated by L2 prefetchers. 

PF_L3_DATA_RD | 7 Counts the number of data cacheline reads generated by L3 prefetchers. 

PF_L3_RFO 8 Counts the number of RFO requests generated by L3 prefetchers. 

PF_L3_CODE_RD /|9 Counts the number of code reads generated by L3 prefetchers. 

SPLIT_LOCK_UC_ | 10 Counts the number of lock requests that split across two cachelines or are to UC memory. 

LOCK 

STRM_ST 11 Counts the number of streaming store requests electronically. 

Reserved 14:12 | Reserved 

OTHER 15 Any other request that crosses IDI, including I/O. 


The supplier information field listed in Table 18-28. The fields vary across products (according to CPUID signatures) 
and is noted in the description. 


Table 18-28. MSR_OFFCORE_RSP_x Supplier Info Field Definition (CPUID Signature 06_3CH, 06_46H) 
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Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier NO_SUPP 17 No Supplier Information available. 
Info L3_HITM 18 M-state initial lookup stat in L3. 
L3_HITE 19 €-state 
L3_HITS 20 S-state 
Reserved 21 Reserved 
LOCAL 22 Local DRAM Controller. 
Reserved 30:23 Reserved 
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Table 18-29. MSR_OFFCORE_RSP_x Supplier Info Field Definition (CPUID Signature 06_45H) 


Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier NO_SUPP 17 No Supplier Information available. 
Info L3_HITM 18 M-state initial lookup stat in L3. 
L3_HITE 19 E-state 
L3_HITS 20 S-state 
Reserved 21 Reserved 
L4_HIT_LOCAL_L4 22 L4 Cache 
L4_HIT_REMOTE_HOPO_L4 23 L4 Cache 
L4_HIT_REMOTE_HOP1_L4 24 L4 Cache 
L4_HIT_REMOTE_HOP2P_L4 25 L4 Cache 
Reserved 30:26 Reserved 
18.3.6.4.1 Off-core Response Performance Monitoring in Intel Xeon Processors E5 v3 Series 
Table 18-28 lists the supplier information field that apply to Intel Xeon processor E5 v3 series (CPUID signature 
06_3FH). 
Table 18-30. MSR_OFFCORE_RSP_x Supplier Info Field Definition 
Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier NO_SUPP 17 No Supplier Information available. 
Info L3_HITM 18 M-state initial lookup stat in L3. 
L3_HITE 19 E-state 
L3_HITS 20 S-state 
L3_HITF 21 F-state 
LOCAL 22 Local DRAM Controller. 
Reserved 26:23 Reserved 
L3_MISS_REMOTE_HOPO 27 Hop O Remote supplier. 
L3_MISS_REMOTE_HOP1 28 Hop 1 Remote supplier. 
L3_MISS_REMOTE_HOP2P 29 Hop 2 or more Remote supplier. 
Reserved 30 Reserved 
18.3.6.5 Performance Monitoring and Intel® TSX 


Chapter 16 of Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 describes the details of 
Intel® Transactional Synchronization Extensions (Intel® TSX). This section describes performance monitoring 
support for Intel TSX. 


If a processor supports Intel TSX, the core PMU enhances its IA32_PERFEVTSELx MSR with two additional bit fields 
for event filtering. Support for Intel TSX is indicated by either (a) CPUID.(EAX=7, ECX=0):RTM[bit 11]=1, or (b) if 
CPUID.07H.EBX.HLE [bit 4] = 1. The TSX-enhanced layout of IA32_PERFEVTSELx is shown in Figure 18-34. The 

two additional bit fields are: 
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° IN_TX (bit 32): When set, the counter will only include counts that occurred inside a transactional region, 
regardless of whether that region was aborted or committed. This bit may only be set if the processor supports 
HLE or RTM. 


¢ IN_TXCP (bit 33): When set, the counter will not include counts that occurred inside of an aborted transac- 
tional region. This bit may only be set if the processor supports HLE or RTM. This bit may only be set for 
IA32_PERFEVTSEL2. 


When the IA32_PERFEVTSELx MSR is programmed with both IN_TX=0 and IN_TXCP=0 on a processor that 
supports Intel TSX, the result in a counter may include detectable conditions associated with a transaction code 
region for its aborted execution (if any) and completed execution. 


In the initial implementation, software may need to take pre-caution when using the IN_TXCP bit. See Table 2-29. 


63 34 31 24 2322 21 2019 18171615 oy 0 


> 


Counter Mask]! | - Pl-lolZl,.. 
(CMASK) i Nly| Nic E S = Unit Mask (UMASK) Event Select 


Reserved 


|_______________ LSR—User Mode 
OS—Operating system mode 

E—Edge detect 
PC—Pin control 
INT—APIC interrupt enable 
ANY—Any Thread 
EN—Enable counters 
INV—lInvert counter mask 
IN_TX—In Trans. Rgn 
IN_TXCP—In Tx exclude abort (PERFEVTSEL2 Only) 


Figure 18-34. Layout of IA32_PERFEVTSELx MSRs Supporting Intel TSX 


A common usage of setting IN_TXCP=1 is to capture the number of events that were discarded due to a transac- 
tional abort. With IA32_PMC2 configured to count in such a manner, then when a transactional region aborts, the 
value for that counter is restored to the value it had prior to the aborted transactional region. As a result, any 
updates performed to the counter during the aborted transactional region are discarded. 


On the other hand, setting IN_TX=1 can be used to drill down on the performance characteristics of transactional 
code regions. When a PMCx is configured with the corresponding IA32_PERFEVTSELx.IN_TX=1, only eventing 
conditions that occur inside transactional code regions are propagated to the event logic and reflected in the 
counter result. Eventing conditions specified by IA32_PERFEVTSELx but occurring outside a transactional region 
are discarded. 


Additionally, a number of performance events are solely focused on characterizing the execution of Intel TSX trans- 


actional code, they can be found at: https://perfmon-events.intel.com/. 


18.3.6.5.1 Intel TSX and PEBS Support 


If a PEBS event would have occurred inside a transactional region, then the transactional region first aborts, and 
then the PEBS event is processed. 


Two of the TSX performance monitoring events also support using the PEBS facility to capture additional informa- 
tion. They are: 


¢ HLE_RETIRED.ABORTED (encoding C8H mask 04H), 
¢ RTM_RETIRED.ABORTED (encoding C9H mask 04H). 


A transactional abort (HLE_RETIRED.ABORTED,RTM_RETIRED.ABORTED) can also be programmed to cause PEBS 
events. In this scenario, a PEBS event is processed following the abort. 
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Pending a PEBS record inside of a transactional region will cause a transactional abort. If a PEBS record was pended 
at the time of the abort or on an overflow of the TSX PEBS events listed above, only the following PEBS entries will 
be valid (enumerated by PEBS entry offset B8H bits[33:32] to indicate an HLE abort or an RTM abort): 


° Offset BOH: EventingIP, 
° €6Offset B8H: TX Abort Information 
These fields are set for all PEBS events. 


° Offset O8H (RIP/EIP) corresponds to the instruction following the outermost XACQUIRE in HLE or the first 
instruction of the fallback handler of the outermost XBEGIN instruction in RTM. This is useful to identify the 
aborted transactional region. 


In the case of HLE, an aborted transaction will restart execution deterministically at the start of the HLE region. In 
the case of RTM, an aborted transaction will transfer execution to the RTM fallback handler. 


The layout of the TX Abort Information field is given in Table 18-31. 


Table 18-31. TX Abort Information Field Definition 


Bit Name Offset | Description 

Cycles_Last_TX 31:0 The number of cycles in the last TSX region, regardless of whether that region had aborted or 
committed. 

HLE_Abort 32 If set, the abort information corresponds to an aborted HLE execution 

RTM_Abort 33 If set, the abort information corresponds to an aborted RTM execution 

Instruction_Abort 34 If set, the abort was associated with the instruction corresponding to the eventing IP (offset 
OBOH) within the transactional region. 

Non_Instruction_Abort 35 If set, the instruction corresponding to the eventing IP may not necessarily be related to the 
transactional abort. 

Retry 36 If set, retrying the transactional execution may have succeeded. 

Data_Conflict 37 If set, another logical processor conflicted with a memory address that was part of the 
transactional region that aborted. 

Capacity Writes 38 If set, the transactional region aborted due to exceeding resources for transactional writes. 

Capacity Reads 39 If set, the transactional region aborted due to exceeding resources for transactional reads. 

Reserved 63:40 | Reserved 


18.3.6.6 Uncore Performance Monitoring Facilities in the 4th Generation Intel® Core” Processors 


The uncore sub-system in the 4th Generation Intel® Core™ processors provides its own performance monitoring 
facility. The uncore PMU facility provides dedicated MSRs to select uncore performance monitoring events in a 
similar manner as those described in Section 18.3.4.6. 


The ARB unit and each C-Box provide local pairs of event select MSR and counter register. The layout of the event 
select MSRs in the C-Boxes are identical as shown in Figure 18-32. 


At the uncore domain level, there is a master set of control MSRs that centrally manages all the performance moni- 
toring facility of uncore units. Figure 18-33 shows the layout of the uncore domain global control. 


Additionally, there is also a fixed counter, counting uncore clockticks, for the uncore domain. Table 18-19 summa- 
rizes the number MSRs for uncore PMU for each box. 
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Table 18-32. Uncore PMU MSR Summary 


Counters per | Counter General Global 
Box # of Boxes Box Width Purpose Enable Comment 
C-Box SKU specific 2 44 Yes Per-box Up to 4, seeTable 2-21 
MSR_UNC_CBO_CONFIG 
ARB 1 2 44 Yes Uncore 
Fixed Counter | N.A. NA. 48 No Uncore 


The uncore performance events for the C-Box and ARB units can be found at: https://perfmon-events.intel.com/. 


18.3.6.7._ Intel® Xeon® Processor €5 v3 Family Uncore Performance Monitoring Facility 


Details of the uncore performance monitoring facility of Intel Xeon Processor E5 v3 families are available in “Intel® 
Xeon® Processor E5 v3 Uncore Performance Monitoring Programming Reference Manual”. The MSR-based uncore 


PMU interfaces are listed in Table 2-33. 


18.3.7. 5th Generation Intel® Core™ Processor and Intel® Core™ M Processor Performance 
Monitoring Facility 
The 5th Generation Intel® Core™ processor and the Intel® Core™ M processor families are based on the Broadwell 


microarchitecture. The core PMU supports architectural performance monitoring capability with version ID 3 (see 
Section 18.2.3) and a host of non-architectural monitoring capabilities. 


Architectural performance monitoring version 3 capabilities are described in Section 18.2.3. 


The core PMU has the same capability as those described in Section 18.3.6. IA32_PERF_GLOBAL_STATUS provide 
a bit indicator (bit 55) for PMI handler to distinguish PMI due to output buffer overflow condition due to accumu- 
lating packet data from Intel Processor Trace. 


636261 55 35 34.33 3231 8 


CondChgd _| 
Ovf_Buffer 


Ovf_UncorePMU——— 
Trace_ToPA_PMI 
FIXED_CTR2 Overflow (RO) 
FIXED_CTR1 Overflow (RO) 
FIXED_CTRO Overflow (RO) 
PMC7_OVF (RO, If PMC7 present) 
PMC6_OVF (RO, If PMC6 present) 
PMC5_OVF (RO, If PMC5 present) 
PMC4_OVF (RO, If PMC4 present) 
PMC3_OVF (RO) 
PMC2_OVF (RO) 
) 
) 


PMC1_OVF (RO 
PMCO_OVF (RO 


Reserved Valid if CPUID.OAH:EAX[15:8] = 8; else reserved 


Figure 18-35. IA32_PERF_GLOBAL_STATUS MSR in Broadwell Microarchitecture 


Details of Intel Processor Trace is described in Chapter 31, “Intel® Processor Trace”. 
IA32_PERF_GLOBAL_OVF_CTRL MSR provide a corresponding reset control bit. 
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636261 55 35 34 33 32 31 8 


ClrCondChgd 
ClrOvfDSBuffer 
ClrOvfUncore 
ClrTraceToPA_PMI 
FIXED_CTR2 ClrOverflow 
FIXED_CTR1 ClrOverflow 
FIXED_CTRO ClrOverflow 
PMC7_ClrOvf (if PMC7 present) 
PMC6_ClrOvf (if PMC6 present) 
) 
) 


PMC5_ClrOvf (if PMC5 present 
PMC4_ClrOvf (if PMC4 present 
PMC3_ClrOvf 


PMC2_ClrOvf 
PMC1_ClrOvf 


PMCO_ClrOvf 


Valid if CPUID.OAH:EAX[15:8] = 8; else reserved 


Reserved 


Figure 18-36. IA32_PERF_GLOBAL_OVF_CTRL MSR in Broadwell microarchitecture 


The specifics of non-architectural performance events can be found at: https://perfmon-events.intel.com/. 


18.3.8 6th Generation, 7th Generation and 8th Generation Intel® Core™” Processor 
Performance Monitoring Facility 


The 6th generation Intel® Core™ processor is based on the Skylake microarchitecture. The 7th generation Intel® 
Core™ processor is based on the Kaby Lake microarchitecture. The 8th generation Intel® Core™ processor is based 
on the Coffee Lake microarchitecture. For these microarchitectures, the core PMU supports architectural perfor- 
mance monitoring capability with version ID 4 (see Section 18.2.4) and a host of non-architectural monitoring 
capabilities. 

Architectural performance monitoring version 4 capabilities are described in Section 18.2.4. 

The core PMU’'s capability is similar to those described in Section 18.6.3 through Section 18.3.4.5, with some differ- 
ences and enhancements summarized in Table 18-22. Additionally, the core PMU provides some enhancement to 
support performance monitoring when the target workload contains instruction streams using Intel® Transactional 
Synchronization Extensions (TSX), see Section 18.3.6.5. For details of Intel TSX, see Chapter 16, “Programming with 
Intel® Transactional Synchronization Extensions” of Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 1. 

Performance monitoring result may be affected by side-band activity on processors that support Intel SGX, details 
are described in Chapter 38, “Enclave Code Debug and Profiling”. 
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Table 18-33. Core PMU Comparison 


Box 


Intel® Microarchitecture Code Name 
Skylake, Kaby Lake and Coffee Lake 


Intel® Microarchitecture Code 
Name Haswell and Broadwell 


Comment 


# of Fixed counters per thread 


3 


3 


Use CPUID to 
determine # of 
counters. See 
Section 18.2.1. 


# of general-purpose counters | 8 8 Use CPUID to 

per core determine # of 
counters. See 
Section 18.2.1. 

Counter width (R,W) R:48, W: 32/48 R:48, W: 32/48 See Section 18.2.2. 


# of programmable counters 
per thread 


4 or (8 if a core not shared by two 
threads) 


4 or (8 if a core not shared by two 
threads) 


Use CPUID to 
determine # of 
counters. See 
Section 18.2.1. 


Architectural Perfmon version 


See Section 18.2.4 


PMI Overhead Mitigation 


= Freeze_Perfmon_on_PMI with 
streamlined semantics. 

« Freeze_LBR_on_PMI with 
streamlined semantics. 

» Freeze_while_SMM. 


« Freeze_Perfmon_on_PMI with 
legacy semantics. 

« Freeze_LBR_on_PMI with 
legacy semantics for branch 
profiling. 

« Freeze_while_SMM. 


See Section 17.4.7. 


Legacy semantics 
not supported with 
version 4 or higher. 


Counter and Buffer Overflow 
Status Management 


= Query via 
IA32_PERF_GLOBAL_STATUS 

« Reset via 
IA32_PERF_GLOBAL_STATUS_RESET 

* Set via 
IA32_PERF_GLOBAL_STATUS_SET 


= Query via 
IA32_PERF_GLOBAL_STATUS 

« Reset via 
IA32_PERF_GLOBAL_OVF_CTRL 


See Section 18.2.4. 


IA32_PERF_GLOBAL_STATUS 
Indicators of 
Overflow/Overhead/Interferen 
ce 


Individual counter overflow 
PEBS buffer overflow 
ToPA buffer overflow 
CTR_Frz, LBR_Frz, ASCI 


* Individual counter overflow 

= PEBS buffer overflow 

= ToPA buffer overflow 
(applicable to Broadwell 
microarchitecture) 


See Section 18.2.4. 


Enable control in = CTR_Frz NA See Section 
IA32_PERF_GLOBAL_STATUS |* LBR_Frz 18.2.4.1. 
Perfmon Counter In-Use Query IA32_PERF_GLOBAL_INUSE NA See Section 
Indicator 18.2.4.3. 


Precise Events 


See Table 18-36. 


See Table 18-12. 


IA32_PMC4-PMC7 
do not support 
PEBS. 


PEBS for front end events See Section 18.3.8.1.4. No 
LBR Record Format Encoding 000101b 000100b Section 17.4.8.1 
LBR Size 32 entries 16 entries 


LBR Entry 


From_IP/To_IP/LBR_Info triplet 


From_IP/To_IP pair 


Section 17.12 


LBR Timing 


Yes 


No 


Section 17.12.1 


Call Stack Profiling 


Yes, see Section 17.11 


Yes, see Section 17.11 


Use LBR facility. 


Off-core Response Event 


MSR 1A6H and 1A7H; Extended request 
and response types. 


MSR 1A6H and 1A7H; Extended 
request and response types. 


Intel TSX support for Perfmon 


See Section 18.3.6.5. 


See Section 18.3.6.5. 
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18.3.8.1 Processor Event Based Sampling (PEBS) Facility 


The PEBS facility in the 6th generation, 7th generation and 8th generation Intel Core processors provides a number 
enhancement relative to PEBS in processors based on Haswell/Broadwell microarchitectures. The key components 
and differences of PEBS facility relative to Haswell/Broadwell microarchitecture is summarized in Table 18-34. 


Table 18-34. PEBS Facility Comparison 


Box Intel® Microarchitecture Code | Intel® Microarchitecture Code Comment 
Name Skylake, Kaby Lake Name Haswell and Broadwell 
and Coffee Lake 
Valid IA32_PMCx PMCO-PMC3 PMCO-PMC3 No PEBS on PMC4-PMC7. 
PEBS Buffer Programming Section 18.3.1.1.1 Section 18.3.1.1.1 Unchanged 
IA32_PEBS_ENABLE Layout | Figure 18-15 Figure 18-15 
PEBS-EventingIP Yes Yes 
PEBS record format encoding | 0011b 0010b 
PEBS record layout Table 18-35; enhanced fields | Table 18-24; enhanced fields at 
at offsets 98H- B8H; and TSC | offsets 98H, AOH, A8H, BOH. 
record field at COH. 
Multi-counter PEBS PEBS record 90H resolves the | PEBS record 90H reflects 
resolution eventing counter overflow. IA32_PERF_GLOBAL_STATUS. 
Precise Events See Table 18-36. See Table 18-12. IA32_PMC4-IA32_PMC7 do not 
support PEBS. 
PEBS-PDIR Yes Yes 1A32_PMC1 only. 
PEBS-Load Latency See Section 18.3.4.4.2. See Section 18.3.4.4.2. 
Data Address Profiling Yes Yes 
FrontEnd event support FrontEnd_Retried event and No IA32_PMCO-PMC3 only. 
MSR_PEBS_FRONTEND. 


Only IA32_PMCO through IA32_PMC3 support PEBS. 


NOTES 


Precise events are only valid when the following fields of IA32_PERFEVTSELx are all zero: 
AnyThread, Edge, Invert, CMask. 


In a PMU with PDIR capability, PEBS behavior is unpredictable if IA32_PERFEVTSELx or IA32_PMCx 
is changed for a PEBS-enabled counter while an event is being counted. To avoid this, changes to 
the programming or value of a PEBS-enabled counter should be performed when the counter is 
disabled. 


18.3.8.1.1 PEBS Data Format 


The PEBS record format for the 6th generation, 7th generation and 8th generation Intel Core processors is 
reporting with encoding 0011b in IA32_PERF_CAPABILITIES[11:8]. The lay out is shown in Table 18-35. The PEBS 
record format, along with debug/store area storage format, does not change regardless of whether IA-32e mode is 
active or not. CPUID.01H:ECX.DTES64[bit 2] reports whether the processor's DS storage format support is mode- 
independent. When set, it uses 64-bit DS storage format. 
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Table 18-35. PEBS Record Format for 6th Generation, 7th Generation 
and 8th Generation Intel Core Processor Families 


Byte Offset Field Byte Offset Field 

00H R/EFLAGS 68H R11 

08H R/EIP 70H R12 

10H R/EAX 78H R13 

18H R/EBX 80H R14 

20H R/ECX 88H R15 

28H R/EDX 90H Applicable Counter 

30H R/ESI 98H Data Linear Address 

38H R/EDI AOH Data Source Encoding 

40H R/EBP A8H Latency value (core cycles) 
48H R/ESP BOH EventinglP 

50H R8 B8H TX Abort Information (Section 18.3.6.5.1) 
58H RO COH TSC 

60H R10 


The layout of PEBS records are largely identical to those shown in Table 18-24. 


The PEBS records at offsets 98H, AOH, and ABH record data gathered from three of the PEBS capabilities in prior 
processor generations: load latency facility (Section 18.3.4.4.2), PDIR (Section 18.3.4.4.4), and data address 
profiling (Section 18.3.6.3). 


In the core PMU of the 6th generation, 7th generation and 8th generation Intel Core processors, load latency facility 
and PDIR capabilities and data address profiling are unchanged relative to the 4th generation and 5th generation 
Intel Core processors. Similarly, precise store is replaced by data address profiling. 


With format 0010b, a snapshot of the IA32_PERF_GLOBAL_STATUS may be useful to resolve the situations when 
more than one of IA32_PMICx have been configured to collect PEBS data and two consecutive overflows of the 
PEBS-enabled counters are sufficiently far apart in time. It is also possible for the image at 90H to indicate multiple 
PEBS-enabled counters have overflowed. In the latter scenario, software cannot to correlate the PEBS record entry 
to the multiple overflowed bits. 


With PEBS record format encoding 0011b, offset 90H reports the “applicable counter” field, which is a multi- 
counter PEBS resolution index allowing software to correlate the PEBS record entry with the eventing PEBS over- 
flow when multiple counters are configured to record PEBS records. Additionally, offset COH captures a snapshot of 
the TSC that provides a time line annotation for each PEBS record entry. 


18.3.8.1.2 PEBS Events 


The list of precise events supported for PEBS in the Skylake, Kaby Lake and Coffee Lake microarchitectures is 
shown in Table 18-36. 
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Table 18-36. Precise Events for the Skylake, Kaby Lake and Coffee Lake Microarchitectures 


Event Name Event Select | Sub-event UMask 
INST_RETIRED COH PREC_DIST! 01H 
ALL_CYCLES? 01H 
OTHER_ASSISTS C1H ANY 3FH 
BR_INST_RETIRED C4H CONDITIONAL 01H 
NEAR_CALL 02H 
ALL_BRANCHES 04H 
NEAR_RETURN 08H 
NEAR_TAKEN 20H 
FAR_BRACHES 40H 
BR_MISP_RETIRED C5H CONDITIONAL 01H 
ALL_BRANCHES 04H 
NEAR_TAKEN 20H 
FRONTEND_RETIRED C6H <Programmable2> 01H 
HLE_RETIRED C8H ABORTED 04H 
RTM_RETIRED CSH ABORTED 04H 
MEM_INST_RETIRED@ DOH LOCK_LOADS 21H 
SPLIT_LOADS 41H 
SPLIT_STORES 42H 
ALL_LOADS 81H 
ALL_STORES 82H 
MEM_LOAD_RETIRED* D1H L1_HIT 01H 
L2_HIT 02H 
L3_HIT 04H 
Li_MISS 08H 
L2_MISS 10H 
L3_MISS 20H 
HIT_LFB 40H 
[MEM_LOAD_L3_HIT_RETIRED? [D2H ~~ +=|XSNP-MISS  ..~—~—~—~—~—CT J OTH—iti‘CSCS@Y 
XSNP_HIT 02H 
XSNP_HITM 04H 
XSNP_NONE 08H 
NOTES: 


1. Only available on [A32_PMC1. 

2. INST_RETIRED.ALL_CYCLES is configured with additional parameters of cmask = 10 and INV = 1 
3. Subevents are specified using MSR_PEBS_FRONTEND, see Section 18.3.8.2 

4. Instruction with at least one load uop experiencing the condition specified in the UMask. 


18.3.8.1.3 Data Address Profiling 


The PEBS Data address profiling on the 6th generation, 7th generation and 8th generation Intel Core processors is 
largely unchanged from the prior generation. When the DataLA facility is enabled, the relevant information written 
into a PEBS record affects entries at offsets 98H, AOH and A8H, as shown in Table 18-26. 
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Table 18-37. Layout of Data Linear Address Information In PEBS Record 


Field Offset | Description 
Data Linear 98H The linear address of the load or the destination of the store. 
Address 


Store Status AOH * DCU Hit (Bit 0): The store hit the data cache closest to the core (L1 cache) if this bit is set, otherwise 

the store missed the data cache. This information is valid only for the following store events: 
UOPS_RETIRED.ALL (if store is tagged), 
MEM_INST_RETIRED.STLB_MISS_STORES, 
MEM_INST_RETIRED.ALL_STORES, 
MEM_INST_RETIRED.SPLIT_STORES. 

« Other bits are zero. 


Reserved A8H Always zero. 


18.3.8.1.4 PEBS Facility for Front End Events 


In the 6th generation, 7th generation and 8th generation Intel Core processors, the PEBS facility has been 
extended to allow capturing PEBS data for some microarchitectural conditions related to front end events. The fron- 
tend microarchitectural conditions supported by PEBS requires the following interfaces: 


¢ The IA32_PERFEVTSELx MSR must select “FrontEnd_Retired” (C6H) in the EventSelect field (bits 7:0) and 
umask = 01H, 


¢ The “FRONTEND_RETIRED” event employs a new MSR, MSR_PEBS_FRONTEND, to specify the supported 
frontend event details, see Table 18-38. 


° Program the PEBS_EN_PMCx field of IA32_PEBS_ENABLE MSR as required. 
Note the AnyThread field of IA32_PERFEVTSELx is ignored by the processor for the “FRONTEND_RETIRED” event. 


The sub-event encodings supported by MSR_PEBS_FRONTEND.EVTSEL is given in Table 18-38. 


Table 18-38. Front€nd_Retired Sub-Event Encodings Supported by MSR_PEBS_FRONTEND.EVTSEL 


Sub-Event Name EVTSEL | Description 


DSB_MISS 11H Retired Instructions which experienced decode stream buffer (DSB) miss. 

L1I_MISS 12H The fetch of retired Instructions which experienced Instruction L1 Cache true miss!. Additional 
requests to the same cache line as an in-flight L11 cache miss will not be counted. 

L2_MISS 13H The fetch of retired Instructions which experienced L2 Cache true miss. Additional requests to the 
same cache line as an in-flight MLC cache miss will not be counted. 

ITLB_MISS 14H The fetch of retired Instructions which experienced ITLB true miss. Additional requests to the same 
cache line as an in-flight ITLB miss will not be counted. 

STLB_MISS 15H The fetch of retired Instructions which experienced STLB true miss. Additional requests to the 
same cache line as an in-flight STLB miss will not be counted. 

IDQ_READ_BUBBLES | 6H An IDQ read bubble is defined as any one of the 4 allocation slots of IDQ that is not filled by the 


front-end on any cycle where there is no back end stall. Using the threshold and latency fields in 
MSR_PEBS_FRONTEND allows counting of IDQ read bubbles of various magnitude and duration. 
Latency controls the number of cycles and Threshold controls the number of allocation slots that 
contain bubbles. 


The event counts if and only if a sequence of at least FE_LATENCY consecutive cycles contain at 
least FE_TRESHOLD number of bubbles each. 


NOTES: 
1. A true miss is the first miss for a cacheline/page (excluding secondary misses that fall into same cacheline/page). 
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The layout of MSR_PEBS_FRONTEND is given in Table 18-39. 


Table 18-39. MSR_PEBS_FRONTEND Layout 


Bit Name Offset | Description 

EVTSEL 7:0 Encodes the sub-event within FrontEnd_Retired that can use PEBS facility, see Table 18-38. 

IDQ_Bubble_Length 19:8 Specifies the threshold of continuously elapsed cycles for the specified width of bubbles when 
counting IDQ_READ_BUBBLES event. 

IDQ_Bubble_Width 22:20 | Specifies the threshold of simultaneous bubbles when counting IDQ_READ_BUBBLES event. 

Reserved 63:23 | Reserved 


18.3.8.1.5 FRONTEND_RETIRED 


The FRONTEND_RETIRED event is designed to help software developers identify exact instructions that caused 
front-end issues. There are some instances in which the event will, by design, the under-counting scenarios include 
the following: 


The event counts only retired (non-speculative) front-end events, i.e. events from just true program execution 
path are counted. 


The event will count once per cacheline (at most). If a cacheline contains multiple instructions which caused 
front-end misses, the count will be only 1 for that line. 


If the multibyte sequence of an instruction spans across two cachelines and causes a miss it will be recorded 
once. If there were additional misses in the second cacheline, they will not be counted separately. 


If a multi-uop instruction exceeds the allocation width of one cycle, the bubbles associated with these uops will 
be counted once per that instruction. 


If 2 instructions are fused (macro-fusion), and either of them or both cause front-end misses, it will be counted 
once for the fused instruction. 


If a front-end (miss) event occurs outside instruction boundary (e.g. due to processor handling of architectural 
event), it may be reported for the next instruction to retire. 


18.3.8.2 |Off-core Response Performance Monitoring 


The core PMU facility to collect off-core response events are similar to those described in Section 18.3.4.5. Each 
event code for off-core response monitoring requires programming an associated configuration MSR, 
MSR_OFFCORE_RSP_x. Software must program MSR_OFFCORE_RSP_.x according to: 


Transaction request type encoding (bits 15:0): see Table 18-40. 
Supplier information (bits 29:16): see Table 18-41. 
Snoop response information (bits 37:30): see Table 18-42. 
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Table 18-40. MSR_OFFCORE_RSP_x Request_Type Definition (Skylake, Kaby Lake 
and Coffee Lake Microarchitectures) 


Bit Name Offset | Description 


DMND_DATA_RD | 0 Counts the number of demand data reads and page table entry cacheline reads. Does not count hw or 
sw prefetches. 


DMND_RFO 1 Counts the number of demand reads for ownership (RFO) requests generated by a write to data 
cacheline. Does not count L2 RFO prefetches. 

DMND_IFETCH 2 Counts the number of demand instruction cacheline reads and L1 instruction cacheline prefetches. 

Reserved 14:3 Reserved 

OTHER 15 Counts miscellaneous requests, such as I/O and un-cacheable accesses. 


Table 18-41 lists the supplier information field that applies to 6th generation, 7th generation and 8th generation 
Intel Core processors. (6th generation Intel Core processor CPUID signatures: 06_4EH, 06_5EH; 7th generation 
and 8th generation Intel Core processor CPUID signatures: 06_8EH, 06_9EH). 


Table 18-41. MSR_OFFCORE_RSP_x Supplier Info Field Definition 
(CPUID Signatures 06_4EH, 06_5EH and 06_8EH, 06_9EH) 


Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier NO_SUPP 17 No Supplier Information available. 
Info L3_HITM 18 M-state initial lookup stat in L3. 
L3_HITE 19 €-state 
L3_HITS 20 S-state 
Reserved 21 Reserved 
L4_HIT 22 L4 Cache (if L4 is present in the processor). 
Reserved 25:23 Reserved 
DRAM 26 Local Node 
Reserved 29:27 Reserved 
SPL_HIT 30 L4 cache super line hit (if L4 is present in the processor). 


Table 18-42 lists the snoop information field that apply to processors with CPUID signatures 06_4EH, 06_5EH, 
06_8EH, 06_9E, and 06_55H. 
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Table 18-42. MSR_OFFCORE_RSP_x Snoop Info Field Definition 
(CPUID Signatures 06_4EH, 06_5EH, 06_8EH, 06_9E and 06_55H) 


Subtype Bit Name 


Offset 


Description 


Snoop Info | SPL_HIT 


30 


L4 cache super line hit (if L4 is present in the processor). 


SNOOP_NONE 


31 


No details on snoop-related information. 


SNOOP_NOT_NEEDED 


32 


No snoop was needed to satisfy the request. 


SNOOP_MISS 


33 


A snoop was needed and it missed all snooped caches: 
-For LLC Hit, ReslHitl was returned by all cores. 


-For LLC Miss, Rspl was returned by all sockets and data was returned from 
DRAM. 


SNOOP_HIT_NO_FWD 


34 


A snoop was needed and it hits in at least one snooped cache. Hit denotes a 
cache-line was valid before snoop effect. This includes: 


-Snoop Hit w/ Invalidation (LLC Hit, RFO). 

-Snoop Hit, Left Shared (LLC Hit/Miss, IFetch/Data_RD). 

-Snoop Hit w/ Invalidation and No Forward (LLC Miss, RFO Hit S). 
In the LLC Miss case, data is returned from DRAM. 


SNOOP_HIT_WITH_FWD 


35 


A snoop was needed and data was forwarded from a remote socket. This 
includes: 


-Snoop Forward Clean, Left Shared (LLC Hit/Miss, IFetch/Data_RD/RFT). 


SNOOP_HITM 


SNOOP_NON_DRAM 


36 


37 


A snoop was needed and it HitM-ed in local or remote cache. HitM denotes a 
cache-line was in modified state before effect as a results of snoop. This 
includes: 


-Snoop HitM w/ WB (LLC miss, IFetch/Data_RD). 
-Snoop Forward Modified w/ Invalidation (LLC Hit/Miss, RFO). 
-Snoop MtoS (LLC Hit, IFetch/Data_RD). 


Target was non-DRAM system address. This includes MMIO transactions. 


18.3.8.2.1 


Off-core Response Performance Monitoring for the Intel® Xeon® Processor Scalable Family 


The following tables list the requestor and supplier information fields that apply to the Intel® Xeon® Processor 
Scalable Family. 


Transaction request type encoding (bits 15:0): see Table 18-43. 
Supplier information (bits 29:16): see Table 18-44. 
Supplier information (bits 29:16) with support for Intel® Optane™ DC Persistent Memory support: see 


Table 18-45. 


Snoop response information has not been changed and is the same as in (bits 37:30): see Table 18-42. 
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Table 18-43. MSR_OFFCORE_RSP_x Request_Type Definition (Intel® Xeon® Processor Scalable Family) 


Bit Name Offset | Description 

DEMAND_DATA_RD 0 Counts the number of demand data reads and page table entry cacheline reads. Does not count 
hw or sw prefetches. 

DEMAND_RFO 1 Counts the number of demand reads for ownership (RFO) requests generated by a write to data 
cacheline. Does not count L2 RFO prefetches. 

DEMAND_CODE_RD 2 Counts the number of demand instruction cacheline reads and L1 instruction cacheline 
prefetches. 

Reserved 3 Reserved. 

PF_L2_DATA_RD 4 Counts the number of prefetch data reads into L2. 

PF_L2_RFO 5 Counts the number of RFO Requests generated by the MLC prefetches to L2. 

Reserved 6 Reserved. 

PF_L3_DATA_RD 7 Counts the number of MLC data read prefetches into L3. 

PF_L3_RFO 8 Counts the number of RFO requests generated by MLC prefetches to L3. 

Reserved 9 Reserved. 

PF_L1D_AND_SW 10 Counts data cacheline reads generated by hardware L1 data cache prefetcher or software 
prefetch requests. 

Reserved 14:11. | Reserved. 

OTHER 15 Counts miscellaneous requests, such as I/O and un-cacheable accesses. 


Table 18-44 lists the supplier information field that applies to the Intel Xeon Processor Scalable Family (CPUID 


signature: 06_55H). 


Table 18-44. MSR_OFFCORE_RSP_x Supplier Info Field Definition (CPUID Signature 06_55H) 


Subtype Bit Name Offset Description 

Common Any 16 Catch all value for any response types. 

Supplier SUPPLIER_NONE 17 No Supplier Information available. 

Info L3_HIT_M 18 M-state initial lookup stat in L3. 
L3_HIT_E 19 E-state 
L3_HIT_S 20 S-state 
L3_HIT_F 21 F-state 
Reserved 25:22 Reserved 
L3_MISS_LOCAL_DRAM 26 L3 Miss: local home requests that missed the L3 cache and were 

serviced by local DRAM. 

L3_MISS_REMOTE_HOPO_DRAM 27 Hop O Remote supplier. 
L3_MISS_REMOTE_HOP1_DRAM 28 Hop 1 Remote supplier. 
L3_MISS_REMOTE_HOP2P_DRAM 29 Hop 2 or more Remote supplier. 
Reserved 30 Reserved 


Table 18-45 lists the supplier information field that applies to the Intel Xeon Processor Scalable Family (CPUID 
signature: 06_55H, Steppings Ox5H - OxFH). 
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Table 18-45. MSR_OFFCORE_RSP_x Supplier Info Field Definition 
(CPUID Signature 06_55H, Steppings 0x5H - OxFH) 


Subtype Bit Name Offset Description 
Common Any 16 Catch all value for any response types. 
Supplier SUPPLIER_NONE 17 No Supplier Information available. 
Info L3_HIT_M 18 M-state initial lookup stat in L3. 
L3_HIT_E 19 E-state 
L3_HIT_S 20 S-state 
L3_HIT_F 21 F-state 
LOCAL_PMM 22 Local home requests that were serviced by local PMM. 
REMOTE_HOPO_PMM 23 Hop O Remote supplier. 
REMOTE_HOP1_PMM 24 Hop 1 Remote supplier. 
REMOTE_HOP2P_PMM 25 Hop 2 or more Remote supplier. 
L3_MISS_LOCAL_DRAM 26 L3 Miss: Local home requests that missed the L3 cache and were 
serviced by local DRAM. 
L3_MISS_REMOTE_HOPO_DRAM 27 Hop O Remote supplier. 
L3_MISS_REMOTE_HOP1_DRAM 28 Hop 1 Remote supplier. 
L3_MISS_REMOTE_HOP2P_DRAM 29 Hop 2 or more Remote supplier. 
Reserved 30 Reserved 
18.3.8.3. Uncore Performance Monitoring Facilities on Intel® Core™ Processors Based on Cannon Lake 


Microarchitecture 


Cannon Lake microarchitecture introduces LLC support of up to six processor cores. To support six processor cores 
and eight LLC slices, existing MSRs have been rearranged and new CBo MSRs have been added. Uncore perfor- 
mance monitoring software drivers from prior generations of Intel Core processors will need to update the MSR 
addresses. The new MSRs and updated MSR addresses have been added to the Uncore PMU listing in Section 
2.17.2, “MSRs Specific to 8th Generation Intel® Core™ i3 Processors” in Intel® 64 and IA-32 Architectures Soft- 
ware Developer’s Manual, Volume 4. 


18.3.9 10th Generation Intel® Core™ Processor Performance Monitoring Facility 


The 10th generation Intel® Core™ processor is based on Ice Lake microarchitecture. For this microarchitecture, 
the core PMU supports architectural performance monitoring capability with version Id 5 (see Section 18.2.5) and 
a host of non-architectural monitoring capabilities. 


The core PMU's capability is similar to those described in Section 18.3.1 through Section 18.3.8, with some differ- 
ences and enhancements summarized in Table 18-46. 


Table 18-46. PEBS Facility Comparison 


Box Ice Lake Microarchitecture | Skylake, Kaby Lake and Coffee Comment 
Lake Microarchitectures 
Architectural Perfmon 5 4 See Section 18.2.5. 
version 
PEBS: Basic functionality Yes Yes See Section 18.3.9.1. 
PEBS record format encoding | 0100b 0011b See Section 18.6.2.4.2. 
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Table 18-46. PEBS Facility Comparison 


Box 


Ice Lake Microarchitecture 


Skylake, Kaby Lake and Coffee 
Lake Microarchitectures 


Comment 


Extended PEBS 


PEBS is extended to all Fixed 
and General Purpose counters 
and to all performance 
monitoring events. 


No 


See Section 18.9.1. 


Adaptive PEBS Yes No See Section 18.9.2. 
Performance Metrics Yes (4) No See Section 18.3.9.3. 
PEBS-PDIR IA32_FIXEDO only IA32_PMC1 only. 
(Corresponding counter control 
MSRs must be enabled.) 
18.3.9.1 Processor Event Based Sampling (PEBS) Facility 


The PEBS facility in the 10th generation Intel Core processors provides a number of enhancements relative to PEBS 
in processors based on the Skylake, Kaby Lake, and Coffee Lake microarchitectures. Enhancement of PEBS facility 
with Extended PEBS and Adaptive PEBS features are described in detail in Section 18.9. 


18.3.9.2 Off-core Response Performance Monitoring 


The core PMU facility to collect off-core response events are similar to those described in Section 18.3.4.5. Each 
event code for off-core response monitoring requires programming an associated configuration MSR, 
MSR_OFFCORE_RSP_x. Software must program MSR_OFFCORE_RSP_x according to: 


° Transaction request type encoding (bits 15:0): see Table 18-[N1]. 
¢ Response type encoding (bits 16-37) of 
— Supplier information: see Table [18-N2]. 
— Snoop response information: see Table [18-N3]. 
e All transactions are tracked at cacheline granularity except some in request type OTHER. 
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Table 18-47. MSR_OFFCORE_RSP_x Request_Type Definition 
(Future Processors Based on Ice Lake Microarchitecture) 


Bit Name Offset Description 
DEMAND_DATA_RD 0 Counts demand data and page table entry reads. 
DEMAND_RFO 1 Counts demand read (RFO) and software prefetches (PREFETCHW) for exclusive ownership 
in anticipation of a write. 
DEMAND_CODE_RD 2 Counts demand instruction fetches and instruction prefetches targeting the L1 instruction 
cache. 
Reserved Reserved 
HWPF_L2_DATA_RD Counts hardware generated data read prefetches targeting the L2 cache. 
HWPF_L2_RFO Counts hardware generated prefetches for exclusive ownership (RFO) targeting the L2 
cache. 
Reserved 6 Reserved 
HWPF_L3 9:7 and 13! | Counts hardware generated prefetches of any type targeting the L3 cache. 
HWPF_L1D_AND_SWPF | 10 Counts hardware generated data read prefetches targeting the L1 data cache and the 
following software prefetches (PREFETCHNTA, PREFETCHTO/1/2). 
STREAMING_WR 11 Counts streaming stores. 
Reserved 12 Reserved 
Reserved 14 Reserved 
OTHER 15 Counts miscellaneous requests, such as I/O and un-cacheable accesses. 
NOTES: 


1. All bits need to be set to 1 to count this type. 


Ice Lake microarchitecture has added a new category of Response subtype, called a Combined Response Info. To 
count a feature in this type, all the bits specified must be set to 1. 


A valid response type must be a non-zero value of the following expression: 

Any | ['OR' of Combined Response Info Bits | [('OR' of Supplier Info Bits) & (‘'OR' of Snoop Info Bits)]] 

If "ANY" bit[16] is set, other response type bits [17-39] are ignored. 

Table 18-48 lists the supplier information field that applies to processors based on Ice Lake microarchitecture. 


Table 18-48. MSR_OFFCORE_RSP_.x Supplier Info Field Definition 
(Processors Based on Ice Lake Microarchitecture) 


Subtype | Bit Name Offset Description 

Common _ | Any 16 Catch all value for any response types. 

Combined | DRAM 26, 31, 32! Requests that are satisfied by DRAM. 

os NON_DRAM | 26, 37! Requests that are satisfied by a NON_DRAM system component. This includes 

ae MMIO transactions. 

L3_MISS 22, 23, 24, 25, 26,27, | Requests that were not supplied by the L3 Cache. The event includes some 

28, 29, 30, 31, 32, 33, | currently reserved bits in anticipation of future memory designs. 
34, 35, 36, 37! 

Supplier | L3_HIT 18,19, 20! Requests that hit in L3 cache. Depending on the snoop response the L3 cache 

Info may have retrieved the cacheline from another core’s cache. 

Reserved 17, 21:25, 27:29 Reserved. 

NOTES: 


1. All bits need to be set to 1 to count this type. 
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Table 18-49 lists the snoop information field that applies to processors based on Ice Lake microarchitecture. 


Table 18-49. MSR_OFFCORE_RSP_x Snoop Info Field Definition 


(Processors Based on Ice Lake Microarchitecture) 


Subtype | Bit Name Offset Description 

Snoop Reserved 30 Reserved. 

Info SNOOP_NOT_NEEDED 32 No snoop was needed to satisfy the request. 
SNOOP_MISS 33 A snoop was sent and none of the snooped caches contained the cacheline. 
SNOOP_HIT_NO_FWD 34 A snoop was sent and hit in at least one snooped cache. The unmodified 

cacheline was not forwarded back, because the L3 already has a valid copy. 

Reserved 35 Reserved. 
SNOOP_HITM 36 A snoop was sent and the cacheline was found modified in another core’s 


caches. The modified cacheline was forwarded to the requesting core. 


18.3.9.3 Performance Metrics 


The Ice Lake core PMU provides built-in support for Top-down Microarchitecture Analysis (TMA) method level 1 
metrics. These metrics are always available to cross-validate performance observations, freeing general purpose 
counters to count other events in high counter utilization scenarios. For more details about the method, refer to 
Top-Down Analysis Method chapter (Appendix B.1) of the Inte/® 64 and IA-32 Architectures Optimization Refer- 


ence Manual. 


A new MSR called MSR_PERF_METRICS reports the metrics directly. Software can check (and/or expose to its 
guests) the availability of the PERF_METRICS feature using IA32_PERF_CAPABILITIES.PERF_METRICS_AVAILABLE 


(bit 15). 


Reserved 


Backend Bound 


Frontend Bound 


Bad Speculation 


Retiring 


31 


v4 


55 


47 


MSR_PERF_METRICS 
Address: 329H 
Scope: Thread 


Reset value : 0x00000000 .00000000 


39 


This register exposes the four TMA Level 1 metrics. The lower 32 bits are divided into four 8-bit fields, as shown by 


Figure 18-37. MSR_PERF_METRICS Definition 


the above figure, each of which is an integer fraction of 255. 
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To support built-in performance metrics, new bits have been added to the following MSRs: 


® JA32_PERF_GLOBAL_CTRL. EN_PERF_METRICS[48]: If this bit is set and fixed counter 3 is enabled, built-in 
performance metrics are enabled. 


® JA32_PERF_GLOBAL_STATUS_SET. SET_OVF_PERF_METRICS[48]: If this bit is set, it will set the status bit in 
the IA32_PERF_GLOBAL_STATUS register for PERF_METRICS. 


e JA32_PERF_GLOBAL_STATUS_RESET. RESET_OVF_PERF_METRICS[48]: If this bit is set, it will clear the status 
bit in the IA32_PERF_GLOBAL_STATUS register for PERF_METRICS. 


° JA32_PERF_GLOBAL_STATUS. OVF_PERF_METRICS[48]: If this bit is set, it indicates that a PERF_METRICS- 
related resource has overflowed and a PMI is triggered‘. If this bit is clear, no such overflow has occurred. 


NOTE 


Software has to synchronize, e.g., re-start, fixed counter 3 as well as PERF_METRICS when either 
bit 35 or 48 in IA32_PERF_GLOBAL_STATUS is set. Otherwise, PERF_METRICS may return 
undefined values. 


The values in MSR_PERF_METRICS are derived from fixed counter 3. Software should start both registers, 
PERF_METRICS and fixed counter 3, from zero. Additionally, software is recommended to periodically clear both 
registers in order to maintain accurate measurements for certain scenarios that involve sampling metrics at high 
rates. 


In order to save/restore PERF_METRICS, software should follow these guidelines: 
¢ PERF_METRICS and fixed counter 3 should be saved and restored together. 


° To ensure that PERF_METRICS and fixed counter 3 remain synchronized, both should be disabled during both 
save and restore. Software should enable/disable them atomically, with a single write to 
IA32_PERF_GLOBAL_CTRL to set/clear both EN_PERF_METRICS[bit 48] and EN_FIXED_CTR3[bit 35]. 


* On state restore, fixed counter 3 must be restored before PERF_METRICS, otherwise undefined results may be 
observed. 


18.3.10 3rd Generation Intel® Xeon® Scalable Family Performance Monitoring Facility 


The 3rd generation Intel® Xeon® Scalable Family of processors are based on Ice Lake microarchitecture. For this 
microarchitecture, the core PMU supports architectural performance monitoring capability with version Id 5 (see 
Section 18.2.5) and a host of non-architectural monitoring capabilities. 


The core PMU's capability is similar to those described in Section 18.3.1 through Section 18.3.8, with some differ- 
ences and enhancements summarized in Table 18-46. 


18.3.10.1 Processor Event Based Sampling (PEBS) Facility 


The PEBS facility in the 3rd generation Intel Xeon Scalable Family of processors provide a number of enhancements 
relative to PEBS in previous generations of processors. Enhancement of PEBS facility with Extended PEBS and 
Adaptive PEBS features are described in detail in Section 18.9. 


Unlike earlier processors, the 3rd generation Intel Xeon Scalable Family of processors (and later generations) can 
support the recording of PEBS records even if accesses to the linear addresses in the DS buffer management area 
or in the PEBS buffer cause VM exits. When a VM exit occurs, the PEBS record is not lost but instead will “skid” and 
be recorded after the virtual machine is resumed (assuming that the cause of the VM exit is resolved). For precise 
events the guest will observe that the record skid by one event occurrence, while for non-precise events the record 
will skid by one instruction. With this “EPT-friendly” enhancement, a virtual-machine monitor may allow use of 

PEBS by guest software even if EPT does not map all guest-physical memory as present and read/write. It remains 


4. An overflow of fixed counter 3 should normally happen first if software follows Intel’s recommendations. 
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the case that an operating system should establish its paging structures to prevent page faults in those paging 
structures. 


The 3rd generation Intel Xeon Scalable Family of processors based on the Ice Lake microarchitecture introduce 
EPT-friendly PEBS. This allows EPT violations and other VM Exits to be taken on PEBS accesses to the DS Area. See 
Section 18.9.5 for details. 


18.4 PERFORMANCE MONITORING (INTEL® XEON™ PHI PROCESSORS) 


NOTE 


This section also applies to the Intel® Xeon Phi™ Processor 7215, 7285, 7295 Series based on 
Knights Mill microarchitecture. 


18.4.1 Intel® Xeon Phi Processor 7200/5200/3200 Performance Monitoring 


The Intel® Xeon Phi™ processor 7200/5200/3200 series are based on the Knights Landing microarchitecture. The 
performance monitoring capabilities are distributed between its tiles (pair of processor cores) and untile 
(connecting many tiles in a physical processor package). Functional details of the tiles and untile of the Knights 
Landing microarchitecture can be found in Chapter 16 of Inte/® 64 and IA-32 Architectures Optimization Reference 
Manual. 


A complete description of the tile and untile PMU programming interfaces for Intel Xeon Phi processors based on the 
Knights Landing microarchitecture can be found in the Technical Document section at 
http://www. intel.com/content/www/us/en/processors/xeon/xeon-phi-detail.html. 


A tile contains a pair of cores attached to a shared L2 cache and is similar to those found in Intel® Atom™ proces- 
sors based on the Silvermont microarchitecture. The processor provides several new capabilities on top of the 
Silvermont performance monitoring facilities. 


The processor supports architectural performance monitoring capability with version ID 3 (see Section 18.2.3) and 
a host of non-architectural performance monitoring capabilities. The processor provides two general-purpose 
performance counters (IA32_PMCO, IA32_PMC1) and three fixed-function performance counters 
(IA32_FIXED_CTRO, IA32_FIXED_CTR1, IA32_FIXED_CTR2). 


Non-architectural performance monitoring in the processor also uses the IA32_PERFEVTSELx MSR to configure a 
set of non-architecture performance monitoring events to be counted by the corresponding general-purpose 
performance counter. 


The bit fields within each IA32_PERFEVTSELx MSR are defined in Figure 18-6 and described in Section 18.2.1.1 and 
Section 18.2.3. The processor supports AnyThread counting in three architectural performance monitoring events. 


18.4.1.1 €nhancements of Performance Monitoring in the Intel’ Xeon Phi™ processor Tile 


The Intel® Xeon Phi™ processor tile includes the following enhancements to the Silvermont microarchitecture. 


¢ AnyThread support. This facility is limited to following three architectural events: Instructions Retired, Unhalted 
Core Cycles, Unhalted Reference Cycles using IA32_FIXED_CTRO-2 and Unhalted Core Cycles, Unhalted 
Reference Cycles using IA32_PERFEVTSELx. 


¢ PEBS-DLA (Processor Event-Based Sampling-Data Linear Address) fields. The processor provides memory 
address in addition to the Silvermont PEBS record support on select events. The PEBS recording format as 
reported by JA32_PERF_CAPABILITIES [11:8] is 2. 


° Off-core response counting facility. This facility in the processor core allows software to count certain 
transaction responses between the processor tile to subsystems outside the tile (untile). Counting off-core 
response requires additional event qualification configuration facility in conjunction with IA32_PERFEVTSELx. 
Two off-core response MSRs are provided to use in conjunction with specific event codes that must be specified 
with IA32_PERFEVTSELx. Two cores do not share the off-core response MSRs. Knights Landing expands off- 
core response capability to match the processor untile changes. 
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° Average request latency measurement. The off-core response counting facility can be combined to use two 
performance counters to count the occurrences and weighted cycles of transaction requests. This facility is 
updated to match the processor untile changes. 


18.4.1.1.1 Processor Event-Based Sampling 


The processor supports processor event based sampling (PEBS). PEBS is supported using IA32_PMCO (see also 
Section 17.4.9, “BTS and DS Save Area”). 


PEBS uses a debug store mechanism to store a set of architectural state information for the processor. The infor- 
mation provides architectural state of the instruction executed after the instruction that caused the event (See 
Section 18.6.2.4). 


The list of PEBS events supported in the processor is shown in the following table. 


Table 18-50. PEBS Performance Events for the Knights Landing Microarchitecture 


Event Name Event Select Sub-event UMask Data Linear 
Address Support 
BR_INST_RETIRED C4H ALL_BRANCHES OOH No 
JCC 7EH No 
TAKEN_JCC FEH No 
CALL FOH No 
REL_CALL FDH No 
IND_CALL FBH No 
NON_RETURN_IND €BH No 
FAR_BRANCH BFH No 
RETURN F7H No 
BR_MISP_RETIRED C5H ALL_BRANCHES OOH No 
JCC 7EH No 
TAKEN_JCC FEH No 
IND_CALL FBH No 
NON_RETURN_IND €BH No 
RETURN F7H No 
MEM_UOPS_RETIRED 04H L2_HIT_LOADS 02H Yes 
L2_MISS_LOADS 04H Yes 
DLTB_MISS_LOADS 08H Yes 
RECYCLEQ 03H LD_BLOCK_ST_FORWARD 01H Yes 
LD_SPLITS 08H Yes 


The PEBS record format 2 supported by processors based on the Knights Landing microarchitecture is shown in 
Table 18-51, and each field in the PEBS record is 64 bits long. 


Table 18-51. PEBS Record Format for the Knights Landing Microarchitecture 


Byte Offset Field Byte Offset Field 
OOH R/EFLAGS 60H R10 
08H R/EIP 68H R11 
10H R/EAX 70H R12 
18H R/EBX 78H R13 
20H R/ECX 80H R14 
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Table 18-51. PEBS Record Format for the Knights Landing Microarchitecture (Contd.) 


Byte Offset Field Byte Offset Field 

28H R/EDX 88H R15 

30H R/ESI 90H IA32_PERF_GLOBAL_STATUS 
38H R/EDI 98H PSDLA 

40H R/EBP AOH Reserved 

48H R/ESP A8H Reserved 

50H R8 BOH EventingRIP 

58H RQ B8H Reserved 


18.4.1.1.2 Offcore Response Event 


Event number 0B7H support offcore response monitoring using an associated configuration MSR, 
MSR_OFFCORE_RSPO (address 1A6H) in conjunction with umask value 01H or MSR_OFFCORE_RSP1 (address 
1A7H) in conjunction with umask value 02H. Table 18-52 lists the event code, mask value and additional off-core 
configuration MSR that must be programmed to count off-core response events using IA32_PMCx. 


Table 18-52. OffCore Response Event Encoding 


Counter Event code UMask Required Off-core Response MSR 
PMCO-1 B7H 01H MSR_OFFCORE_RSPO (address 1A6H) 
PMCO-1 B7H 02H MSR_OFFCORE_RSP1 (address 1A7H) 


Some of the MSR_OFFCORE_RESP [0,1] register bits are not valid in this processor and their use is reserved. The 
layout of MSR_OFFCORE_RSPO and MSR_OFFCORE_RSP1 registers are defined in Table 18-53. Bits 15:0 specifies 
the request type of a transaction request to the uncore. Bits 30:16 specifies supplier information, bits 37:31 spec- 
ifies snoop response information. 


Additionally, MSR_OFFCORE_RSPO provides bit 38 to enable measurement of average latency of specific type of 
offcore transaction requests using two programmable counter simultaneously, see Section 18.5.2.3 for details. 
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Table 18-53. Bit fields of the MSR_OFFCORE_RESP [0, 1] Registers 


Main Sub-field Bit Name Description 
Request Type 0 DEMAND_DATA_RD Demand cacheable data and L1 prefetch data reads. 
1 DEMAND_RFO Demand cacheable data writes. 
2 DEMAND_CODE_RD Demand code reads and prefetch code reads. 
3 Reserved Reserved. 
4 Reserved Reserved. 
5 PF_L2_RFO L2 data RFO prefetches (includes PREFETCHW instruction). 
6 PF_L2_CODE_RD L2 code HW prefetches. 
7 PARTIAL_READS Partial reads (UC or WC). 
8 PARTIAL_WRITES Partial writes (UC or WT or WP). Valid only for 
OFFCORE_RESP_1 event. Should only be used on PMC1. 
This bit is reserved for OFFCORE_RESP_O event. 
9 UC_CODE_READS UC code reads. 
10 BUS_LOCKS Bus locks and split lock requests. 
11 FULL_STREAMING_STO | Full streaming stores (WC). Valid only for OFFCORE_RESP_1 
RES event. Should only be used on PMC1. This bit is reserved for 
OFFCORE_RESP_O event. 
12 SW_PREFETCH Software prefetches. 
13 PF_L1_DATA_RD L1 data HW prefetches. 
14 PARTIAL_STREAMING_ | Partial streaming stores (WC). Valid only for 
STORES OFFCORE_RESP_1 event. Should only be used on PMC1. 
This bit is reserved for OFFCORE_RESP_O event. 
15 ANY_REQUEST Account for any requests. 
Response Type | Any 16 ANY_RESPONSE Account for any response. 
Data Supply from | 17 NO_SUPP No Supplier Details. 
Untile 
18 Reserved Reserved. 
19 L2_HIT_OTHER_TILE_N | Other tile L2 hit € Near. 
EAR 
20 Reserved Reserved. 
21 MCDRAM_NEAR MCDRAM Local. 
22 MCDRAM_FAR_OR_L2_ | MCDRAM Far or Other tile L2 hit far. 
HIT_OTHER_TILE_FAR 
23 DRAM_NEAR DRAM Local. 
24 DRAM_FAR DRAM Far. 
Data Supply from | 25 L2_HITM_THIS_TILE M-state. 
within same tlle [56° [U2 HITE_THIS_TILE | E-state. 
27 L2_HITS_THIS_TILE S-state. 
28 L2_HITF_THIS_TILE F-state. 
29 Reserved Reserved. 
30 Reserved Reserved. 
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Table 18-53. Bit fields of the MSR_OFFCORE_RESP [0, 1] Registers (Contd.) 


Main Sub-field Bit Name Description 
Snoop Info; Only 31 SNOOP_NONE None of the cores were snooped. 
Valid in case of 32 NO_SNOOP_NEEDED No snoop was needed to satisfy the request. 
Data Supply from 
Untile 33 Reserved Reserved. 
34 Reserved Reserved. 
35 HIT_OTHER_TILE_FWD | Snoop request hit in the other tile with data forwarded. 
36 HITM_OTHER_TILE A snoop was needed and it HitM-ed in other core’s L1 cache. 
HitM denotes a cache-line was in modified state before 
effect as a result of snoop. 
37 NON_DRAM Target was non-DRAM system address. This includes MMIO 
transactions. 
Outstanding Weighted cycles 38 OUTSTANDING (Valid If set, counts total number of weighted cycles of any 
requests only for outstanding offcore requests with data response. Valid only 


MSR_OFFCORE_RESPO. 
Should only be used on 
PMCO. This bit is 
reserved for 
MSR_OFFCORE_RESP1). 


for OFFCORE_RESP_O event. Should only be used on PMCO. 
This bit is reserved for OFFCORE_RESP_1 event. 


18.4.1.1.3 Average Offcore Request Latency Measurement 


Measurement of average latency of offcore transaction requests can be enabled using MSR_OFFCORE_RSPO. [bit 
38] with the choice of request type specified in MSR_OFFCORE_RSPO.[bit 15:0]. 


Refer to Section 18.5.2.3, “Average Offcore Request Latency Measurement,” for typical usage. Note that 
MSR_OFFCORE_RESPx registers are not shared between cores in Knights Landing. This allows one core to measure 
average latency while other core is measuring different offcore response events. 


18.5 


18.5.1 


PERFORMANCE MONITORING (INTEL ATOM® PROCESSORS) 


Performance Monitoring (45 nm and 32 nm Intel Atom® Processors) 


45 nm and 32 nm Intel Atom processors report architectural performance monitoring versionID = 3 (supporting the 
aggregate capabilities of versionID 1, 2, and 3; see Section 18.2.3) and a host of non-architectural monitoring 
capabilities. These 45 nm and 32 nm Intel Atom processors provide two general-purpose performance counters 
(IA32_PMCO, IA32_PMC1) and three fixed-function performance counters (IA32_FIXED_CTRO, IA32_FIXED_CTR1, 
IA32_FIXED_CTR2). 


NOTE 


The number of counters available to software may vary from the number of physical counters 
present on the hardware, because an agent running at a higher privilege level (e.g., a VMM) may 
not expose all counters. CPUID.OAH:EAX[15:8] reports the MSRs available to software; see Section 


18.2.1. 


Non-architectural performance monitoring in Intel Atom processor family uses the IA32_PERFEVTSELx MSR to 
configure a set of non-architecture performance monitoring events to be counted by the corresponding general- 
purpose performance counter. The list of non-architectural performance monitoring events can be found at: 


https://perfmon-events.intel.com/. 
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Architectural and non-architectural performance monitoring events in 45 nm and 32 nm Intel Atom processors 
support thread qualification using bit 21 (AnyThread) of IA32_PERFEVTSELx MSR, i.e., if 
IA32_PERFEVTSELx.AnyThread =1, event counts include monitored conditions due to either logical processors in 
the same processor core. 


The bit fields within each IA32_ PERFEVTSELx MSR are defined in Figure 18-6 and described in Section 18.2.1.1 and 
Section 18.2.3. 


Valid event mask (Umask) bits can be found at: https://perfmon-events.intel.com/. The UMASK field may contain 
sub-fields that provide the same qualifying actions like those listed in Table 18-71, Table 18-72, Table 18-73, and 
Table 18-74. One or more of these sub-fields may apply to specific events on an event-by-event basis. Precise 
Event Based Monitoring is supported using IA32_PMCO (see also Section 17.4.9, “BTS and DS Save Area”). 


18.5.2 Performance Monitoring for Silvermont Microarchitecture 


Intel processors based on the Silvermont microarchitecture report architectural performance monitoring versionID 
= 3 (see Section 18.2.3) and a host of non-architectural monitoring capabilities. Intel processors based on the 
Silvermont microarchitecture provide two general-purpose performance counters (IA32_PMCO, IA32_PMC1) and 
three fixed-function performance counters (IA32_FIXED_CTRO, IA32_FIXED_CTR1, IA32_FIXED_CTR2). Intel 
Atom processors based on the Airmont microarchitecture support the same performance monitoring capabilities as 
those based on the Silvermont microarchitecture. 


Non-architectural performance monitoring in the Silvermont microarchitecture uses the IA32_PERFEVTSELx MSR 
to configure a set of non-architecture performance monitoring events to be counted by the corresponding general- 
purpose performance counter. The list of non-architectural performance monitoring events can be found at: 


https://perfmon-events.intel.com/. 


The bit fields (except bit 21) within each IA32_PERFEVTSELx MSR are defined in Figure 18-6 and described in 
Section 18.2.1.1 and Section 18.2.3. Architectural and non-architectural performance monitoring events in the 
Silvermont microarchitecture ignore the AnyThread qualification regardless of its setting in IA32_PERFEVTSELx 
MSR. 


18.5.2.1 Enhancements of Performance Monitoring in the Processor Core 
The notable enhancements in the monitoring of performance events in the processor core include: 
¢ The width of counter reported by CPUID.OAH:EAX[23:16] is 40 bits. 


°  Off-core response counting facility. This facility in the processor core allows software to count certain 
transaction responses between the processor core to sub-systems outside the processor core (uncore). 
Counting off-core response requires additional event qualification configuration facility in conjunction with 
IA32_PERFEVTSELx. Two off-core response MSRs are provided to use in conjunction with specific event codes 
that must be specified with IA32_PERFEVTSELx. 


e Average request latency measurement. The off-core response counting facility can be combined to use two 
performance counters to count the occurrences and weighted cycles of transaction requests. 


18.5.2.1.1 Processor Event Based Sampling (PEBS) 


In the Silvermont microarchitecture, the PEBS facility can be used with precise events. PEBS is supported using 
IA32_PMCO (see also Section 17.4.9). 


PEBS uses a debug store mechanism to store a set of architectural state information for the processor. The infor- 
mation provides architectural state of the instruction executed after the instruction that caused the event (See 
Section 18.6.2.4). 


The list of precise events supported in the Silvermont microarchitecture is shown in Table 18-54. 
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Table 18-54. PEBS Performance Events for the Silvermont Microarchitecture 


Event Name Event Select Sub-event UMask 
BR_INST_RETIRED C4H ALL_BRANCHES OOH 
JCC 7EH 
TAKEN_JCC FEH 
CALL FOH 
REL_CALL FDH 
IND_CALL FBH 
NON_RETURN_IND €BH 
FAR_BRANCH BFH 
RETURN F7H 
BR_MISP_RETIRED C5SH ALL_BRANCHES OOH 
JCC 7EH 
TAKEN_JCC FEH 
IND_CALL FBH 
NON_RETURN_IND €BH 
RETURN F7H 
MEM_UOPS_RETIRED 04H L2_HIT_LOADS 02H 
L2_MISS_LOADS 04H 
DLTB_MISS_LOADS 08H 
HITM 20H 
REHABQ 03H LD_BLOCK_ST_FORWARD 01H 
LD_SPLITS 08H 


PEBS Record Format The PEBS record format supported by processors based on the Intel Silvermont microarchitec- 
ture is shown in Table 18-55, and each field in the PEBS record is 64 bits long. 


Table 18-55. PEBS Record Format for the Silvermont Microarchitecture 


Byte Offset Field Byte Offset Field 

OOH R/EFLAGS 60H R10 

08H R/EIP 68H R11 

10H R/EAX 70H R12 

18H R/EBX 78H R13 

20H R/ECX 80H R14 

28H R/EDX 88H R15 

30H R/ESI 90H IA32_PERF_GLOBAL_STATUS 
38H R/EDI 98H Reserved 
40H R/EBP AOH Reserved 
48H R/ESP A8H Reserved 
50H R8 BOH EventingRIP 
58H RQ B8H Reserved 
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18.5.2.2 Offcore Response Event 


Event number 0B7H support offcore response monitoring using an associated configuration MSR, 
MSR_OFFCORE_RSPO (address 1A6H) in conjunction with umask value 01H or MSR_OFFCORE_RSP1 (address 
1A7H) in conjunction with umask value 02H. Table 18-56 lists the event code, mask value and additional off-core 
configuration MSR that must be programmed to count off-core response events using IA32_PMCx. 


In the Silvermont microarchitecture, each MSR_OFFCORE_RSPx is shared by two processor cores. 


Table 18-56. OffCore Response Event Encoding 


Counter Event code UMask Required Off-core Response MSR 
PMCO-1 B7H 01H MSR_OFFCORE_RSPO (address 1A6H) 
PMCO-1 B7H 02H MSR_OFFCORE_RSP1 (address 1A7H) 


The layout of MSR_OFFCORE_RSPO and MSR_OFFCORE_RSP1 are shown in Figure 18-38 and Figure 18-39. Bits 
15:0 specifies the request type of a transaction request to the uncore. Bits 30:16 specifies supplier information, 
bits 37:31 specifies snoop response information. 


Additionally, MSR_OFFCORE_RSPO provides bit 38 to enable measurement of average latency of specific type of 
offcore transaction requests using two programmable counter simultaneously, see Section 18.5.2.3 for details. 


63 37 15 1413121110 9876 543 21 0 


See Figure 18-30 


RESPONSE TYPE — Other (R/W) | 


REQUEST TYPE — PARTIAL_STRM_ST (R/W) 
REQUEST TYPE — PF_DATA_RD (R/W) 
REQUEST TYPE — SW_PREFETCH (R/W) 
REQUEST TYPE — STRM_ST (R/W) 
REQUEST TYPE — BUS_LOCKS (R/W) 
REQUEST TYPE — UC_IFETCH (R/W) 
REQUEST TYPE — PARTIAL_WRITE (R/W) 
REQUEST TYPE — PARTIAL_READ (R/W) 
REQUEST TYPE — PF_IFETCH (R/W) 
REQUEST TYPE — PF_RFO (R/W) 
REQUEST TYPE — PF_DATA_RD (R/W) 
REQUEST TYPE — WB (R/W) 
REQUEST TYPE — DMND_IFETCH (R/W) 
REQUEST TYPE — DMND_RFO (R/W) 
REQUEST TYPE — DMND_DATA_RD (R/W) 


Reserved RESET Value — 00000000_00000000H 


Figure 18-38. Request_Type Fields for MSR_OFFCORE_RSPx 


Table 18-57. MSR_OFFCORE_RSPx Request_Type Field Definition 
Bit Name Offset | Description 


DMND_DATA_RD 0 Counts the number of demand and DCU prefetch data reads of full and partial cachelines as well as 
demand data page table entry cacheline reads. Does not count L2 data read prefetches or 
instruction fetches. 


DMND_RFO 1 Counts the number of demand and DCU prefetch reads for ownership (RFO) requests generated by 
a write to data cacheline. Does not count L2 RFO prefetches. 

DMND_IFETCH 2 Counts the number of demand instruction cacheline reads and L1 instruction cacheline prefetches. 

WB 3 Counts the number of writeback (modified to exclusive) transactions. 
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Table 18-57. MSR_OFFCORE_RSPx Request_Type Field Definition (Contd.) 


Bit Name Offset | Description 

PF_DATA_RD 4 Counts the number of data cacheline reads generated by L2 prefetchers. 

PF_RFO 5 Counts the number of RFO requests generated by L2 prefetchers. 

PF_IFETCH 6 Counts the number of code reads generated by L2 prefetchers. 

PARTIAL_READ 7 Counts the number of demand reads of partial cache lines (including UC and WC). 

PARTIAL_WRITE 8 Counts the number of demand RFO requests to write to partial cache lines (includes UC, WT and 
WP) 

UC_IFETCH 9 Counts the number of UC instruction fetches. 

BUS_LOCKS 10 Bus lock and split lock requests 

STRM_ST 11 Streaming store requests 

SW_PREFETCH 12 Counts software prefetch requests 

PF_DATA_RD 13 Counts DCU hardware prefetcher data read requests 

PARTIAL_STRM_ST 14 Streaming store requests 

ANY 15 Any request that crosses IDI, including I/O. 


63 


38 37 36 35 34 33 32 31 22 212019 18 17 16 


AVG LATENCY — ENABLE AVG LATENCY(R/W) 
RESPONSE TYPE — NON_DRAM (R/W) 


RSPNS_SNOOP — HITM (R/W) 


RESERVED 


RSPNS_SNOOP — SNOOP _HIT (R/W) 
RSPNS_SNOOP — SNOOP_MISS (R/W) 


RESERVED 


RSPNS_SNOOP — SNOOP_NONE (R/W) 


RESERVED 


RSPNS_SUPPLIER — L2_HIT (RW) 


RESERVED 


RSPNS_SUPPLIER — ANY (R/W) 


Reserved 


RESET Value — 00000000_00000000H 


Figure 18-39. Response_Supplier and Snoop Info Fields for MSR_OFFCORE_RSPx 


To properly program this extra register, software must set at least one request type bit (Table 18-57) and a valid 
response type pattern (Table 18-58, Table 18-59). Otherwise, the event count reported will be zero. It is permis- 
sible and useful to set multiple request and response type bits in order to obtain various classes of off-core 
response events. Although MSR_OFFCORE_RSPx allow an agent software to program numerous combinations that 
meet the above guideline, not all combinations produce meaningful data. 


Table 18-58. MSR_OFFCORE_RSP_x Response Supplier Info Field Definition 


Subtype Bit Name Offset Description 
Common ANY_RESPONSE 16 Catch all value for any response types. 
Supplier Info | Reserved 17 Reserved 
L2_HIT 18 Cache reference hit L2 in either M/E/S states. 
Reserved 30:19 Reserved 
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To specify a complete offcore response filter, software must properly program bits in the request and response type 
fields. A valid request type must have at least one bit set in the non-reserved bits of 15:0. A valid response type 
must be a non-zero value of the following expression: 


ANY | [COR’ of Supplier Info Bits) & ((OR’ of Snoop Info Bits) ] 
If “ANY” bit is set, the supplier and snoop info bits are ignored. 


Table 18-59. MSR_OFFCORE_RSPx Snoop Info Field Definition 


Subtype | Bit Name Offset | Description 
Snoop SNP_NONE 31 No details on snoop-related information. 
Info 
Reserved 32 Reserved 
SNOOP_MISS 33 Counts the number of snoop misses when L2 misses. 
SNOOP_HIT 34 Counts the number of snoops hit in the other module where no modified copies were 
found. 
Reserved 35 Reserved 
HITM 36 Counts the number of snoops hit in the other module where modified copies were 
found in other core’s L1 cache. 
NON_DRAM 37 Target was non-DRAM system address. This includes MMIO transactions. 
AVG_LATENCY 38 Enable average latency measurement by counting weighted cycles of outstanding 
offcore requests of the request type specified in bits 15:0 and any response (bits 37:16 
cleared to 0). 
This bit is available in MSR_OFFCORE_RESPO. The weighted cycles is accumulated in the 
specified programmable counter IA32_PMCx and the occurrence of specified requests 
are counted in the other programmable counter. 


18.5.2.3 Average Offcore Request Latency Measurement 


Average latency for offcore transactions can be determined by using both MSR_OFFCORE_RSP registers. Using two 
performance monitoring counters, program the two OFFCORE_RESPONSE event encodings into the corresponding 
IA32_PERFEVTSELx MSRs. Count the weighted cycles via MSR_OFFCORE_RSPO by programming a request type in 
MSR_OFFCORE_RSPO.[15:0] and setting MSR_OFFCORE_RSPO.OUTSTANDING[38] to 1, white setting the 
remaining bits to 0. Count the number of requests via MSR_OFFCORE_RSP1 by programming the same request 
type from MSR_OFFCORE_RSPO into MSR_OFFCORE_RSP1[bit 15:0], and setting 
MSR_OFFCORE_RSP1.ANY_RESPONSE[16] = 1, while setting the remaining bits to 0. The average latency can be 
obtained by dividing the value of the IA32_PMCx register that counted weight cycles by the register that counted 
requests. 


18.5.3. Performance Monitoring for Goldmont Microarchitecture 


Intel Atom processors based on the Goldmont microarchitecture report architectural performance monitoring 
versionID = 4 (see Section 18.2.4) and support non-architectural monitoring capabilities described in this section. 


Architectural performance monitoring version 4 capabilities are described in Section 18.2.4. 


The bit fields (except bit 21) within each IA32_PERFEVTSELx MSR are defined in Figure 18-6 and described in 
Section 18.2.1.1 and Section 18.2.3. The Goldmont microarchitecture does not support Hyper-Threading and thus 
architectural and non-architectural performance monitoring events ignore the AnyThread qualification regardless 
of its setting in the IA32_PERFEVTSELx MSR. However, Goldmont does not set the AnyThread deprecation bit 
(CPUID.OAH:EDX[15]). 


The core PMU's capability is similar to that of the Silvermont microarchitecture described in Section 18.5.2 , with 
some differences and enhancements summarized in Table 18-60. 
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Table 18-60. Core PMU Comparison Between the Goldmont and Silvermont Microarchitectures 


Box 


The Goldmont microarchitecture 


The Silvermont microarchitecture 


Comment 


# of Fixed counters per core 


3 


3 


Use CPUID to determine # 
of counters. See Section 
18.2.1. 


# of general-purpose 4 2 Use CPUID to determine # 

counters per core of counters. See Section 
18.2.1. 

Counter width (R,W) R:48, W: 32/48 R:40, W:32 See Section 18.2.2. 

Architectural Performance 4 3 Use CPUID to determine # 


Monitoring version ID 


of counters. See Section 
18.2.1. 


PMI Overhead Mitigation 


= Freeze_Perfmon_on_PMI with 
streamlined semantics. 

" Freeze_LBR_on_PMI with 
streamlined semantics for 
branch profiling. 


* Freeze_Perfmon_on_PMI with 
legacy semantics. 

* Freeze_LBR_on_PMI with legacy 
semantics for branch profiling. 


See Section 17.4.7. 
Legacy semantics not 
supported with version 4 
or higher. 


Counter and Buffer 
Overflow Status 
Management 


" Query via 
|A32_PERF_GLOBAL_STATUS 

= Reset via 
IA32_PERF_GLOBAL_STATUS_R 
ESET 

= Set via 
IA32_PERF_GLOBAL_STATUS_S 
ET 


= Query via 
IA32_PERF_GLOBAL_STATUS 

= Reset via 
IA32_PERF_GLOBAL_OVF_CTRL 


See Section 18.2.4. 


IA32_PERF_GLOBAL_STATU 
S Indicators of 
Overflow/Overhead/Interfer 
ence 


Individual counter overflow 
PEBS buffer overflow 
ToPA buffer overflow 
CTR_Frz, LBR_Frz 


* Individual counter overflow 
= PEBS buffer overflow 


See Section 18.2.4. 


Enable control in * CTR_Frz, No See Section 18.2.4.1. 
IA32_PERF_GLOBAL_STATU |* LBR_Frz 

S 

Perfmon Counter In-Use Query IA32_PERF_GLOBAL_INUSE | No See Section 18.2.4.3. 


Indicator 


Processor Event Based 
Sampling (PEBS) Events 


General-Purpose Counter O only. 
Supports all events (precise and 
non-precise). Precise events are 
listed in Table 18-61. 


See Section 18.5.2.1.1. General- 
Purpose Counter 0 only. Only 
supports precise events (see 
Table 18-54). 


IA32_PMCO only. 


PEBS record format 0011b 0010b 

encoding 

Reduce skid PEBS IA32_PMCO only No 

Data Address Profiling Yes No 

PEBS record layout Table 18-62; enhanced fields at Table 18-55. 
offsets 9OH- 98H; and TSC record 
field at COH. 

PEBS EventingIP Yes Yes 


Off-core Response Event 
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18.5.3.1 Processor Event Based Sampling (PEBS) 


Processor event based sampling (PEBS) on the Goldmont microarchitecture is enhanced over prior generations 
with respect to sampling support of precise events and non-precise events. In the Goldmont microarchitecture, 
PEBS is supported using IA32_PMCO for all events (see Section 17.4.9). 


PEBS uses a debug store mechanism to store a set of architectural state information for the processor at the time 
the sample was generated. 


Precise events work the same way on Goldmont microarchitecture as on the Silvermont microarchitecture. The 
record will be generated after an instruction that causes the event when the counter is already overflowed and will 
capture the architectural state at this point (see Section 18.6.2.4 and Section 17.4.9). The eventingIP in the record 
will indicate the instruction that caused the event. The list of precise events supported in the Goldmont microarchi- 
tecture is shown in Table 18-61. 


In the Goldmont microarchitecture, the PEBS facility also supports the use of non-precise events to record 
processor state information into PEBS records with the same format as with precise events. 


However, a non-precise event may not be attributable to a particular retired instruction or the time of instruction 
execution. When the counter overflows, a PEBS record will be generated at the next opportunity. Consider the 
event ICACHE.HIT. When the counter overflows, the processor is fetching future instructions. The PEBS record will 
be generated at the next opportunity and capture the state at the processor's current retirement point. It is likely 
that the instruction fetch that caused the event to increment was beyond that current retirement point. Other 
examples of non-precise events are CPU_CLK_UNHALTED.CORE_P and HARDWARE_INTERRUPTS.RECEIVED. 
CPU_CLK_UNHALTED.CORE_P will increment each cycle that the processor is awake. When the counter over-flows, 
there may be many instructions in various stages of execution. Additionally, zero, one or multiple instructions may 
be retired the cycle that the counter overflows. HARDWARE_INTERRUPTS.RECEIVED increments independent of 
any instructions being executed. For all non-precise events, the PEBS record will be generated at the next oppor- 
tunity, after the counter has overflowed. The PEBS facility thus allows for identification of the instructions which 
were executing when the event overflowed. 


After generating a record for a non-precise event, the PEBS facility reloads the counter and resumes execution, just 
as is done for precise events. Unlike interrupt-based sampling, which requires an interrupt service routine to collect 
the sample and reload the counter, the PEBS facility can collect samples even when interrupts are masked and 
without using NMI. Since a PEBS record is generated immediately when a counter for a non-precise event is 
enabled, it may also be generated after an overflow is set by an MSR write to IA32_PERF_GLOBAL_STATUS_ SET. 


Table 18-61. Precise Events Supported by the Goldmont Microarchitecture 


Event Name Event Select Sub-event UMask 
LD_BLOCKS 03H DATA_UNKNOWN 01H 
STORE_FORWARD 02H 
4K_ALIAS 04H 
UTLB_MISS 08H 
ALL_BLOCK 10H 
MISALIGN_MEM_REF 13H LOAD_PAGE_SPLIT 02H 
STORE_PAGE_SPLIT 04H 
INST_RETIRED COH ANY OOH 
UOPS_RETITRED C2H ANY OOH 
LD_SPLITSMS 01H 
BR_INST_RETIRED C4H ALL_BRANCHES OOH 
Jcc 7EH 
TAKEN_JCC FEH 
CALL FSH 
REL_CALL FDH 
IND_CALL FBH 
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Table 18-61. Precise Events Supported by the Goldmont Microarchitecture (Contd.) 


Event Name Event Select Sub-event UMask 
NON_RETURN_IND €BH 
FAR_BRANCH BFH 
RETURN F7H 
BR_MISP_RETIRED C5H ALL_BRANCHES OOH 
JCC 7EH 
TAKEN_JCC FEH 
IND_CALL FBH 
NON_RETURN_IND —BH 
RETURN F7H 
MEM_UOPS_RETIRED DOH ALL_LOADS 81H 
ALL_STORES 82H 
ALL 83H 
DLTB_MISS_LOADS 11H 
DLTB_MISS_STORES 12H 
DLTB_MISS 13H 
MEM_LOAD_UOPS_RETIRED D1H L1_HIT 01H 
L2_HIT 02H 
L1_MISS 08H 
L2_MISS 10H 
HITM 20H 
WCB_HIT 40H 
DRAM_HIT 80H 


The PEBS record format supported by processors based on the Intel Goldmont microarchitecture is shown in 
Table 18-62, and each field in the PEBS record is 64 bits long. 


Table 18-62. PEBS Record Format for the Goldmont Microarchitecture 


Byte Offset Field Byte Offset Field 

OOH R/EFLAGS 68H R11 

08H R/EIP 70H R12 

10H R/EAX 78H R13 

18H R/EBX 80H R14 

20H R/ECX 88H R15 

28H R/EDX 90H Applicable Counters 
30H R/ESI 98H Data Linear Address 
38H R/EDI AOH Reserved 

40H R/EBP A8H Reserved 

48H R/ESP BOH EventingRIP 

50H R8 B8H Reserved 

58H RO COH TSC 

60H R10 
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On Goldmont microarchitecture, all 64 bits of architectural registers are written into the PEBS record regardless of 
processor mode. 


With PEBS record format encoding 0011b, offset 90H reports the "Applicable Counter" field, which indicates which 
counters actually requested generating a PEBS record. This allows software to correlate the PEBS record entry 
properly with the instruction that caused the event even when multiple counters are configured to record PEBS 
records and multiple bits are set in the field. Additionally, offset COH captures a snapshot of the TSC that provides 
a time line annotation for each PEBS record entry. 


18.5.3.1.1 PEBS Data Linear Address Profiling 


Goldmont supports the Data Linear Address field introduced in Haswell. It does not support the Data Source 
Encoding or Latency Value fields that are also part of Data Address Profiling; those fields are present in the record 
but are reserved. 


For Goldmont microarchitecture, the Data Linear Address field will record the linear address of memory accesses in 
the previous instruction (e.g. the one that triggered a precise event that caused the PEBS record to be generated). 
Goldmont microarchitecture may record a Data Linear Address for the instruction that caused the event even for 
events not related to memory accesses. This may differ from other microarchitectures. 


18.5.3.1.2 Reduced Skid PEBS 


For precise events, upon triggering a PEBS assist, there will be a finite delay between the time the counter over- 
flows and when the microcode starts to carry out its data collection obligations. The Reduced Skid mechanism miti- 
gates the “skid” problem by providing an early indication of when the counter is about to overflow, allowing the 
machine to more precisely trap on the instruction that actually caused the counter overflow thus greatly reducing 
skid. 


This mechanism is a superset of the PDIR mechanism available in the Sandy Bridge microarchitecture. See Section 
18.3.4.4.4 


In the Goldmont microarchitecture, the mechanism applies to all precise events including, INST_RETIRED, except 
for UOPS_RETIRED. However, the Reduced Skid mechanism is disabled for any counter when the INV, ANY, E, or 
CMASK fields are set. 


With Reduced Skid PEBS, the skid is precisely one event occurrence. Hence if counting INST_RETIRED, PEBS will 
indicate the instruction that follows that which caused the counter to overflow. 


For the Reduced Skid mechanism to operate correctly, the performance monitoring counters should not be recon- 
figured or modified when they are running with PEBS enabled. The counters need to be disabled (e.g. via 
IA32_PERF_GLOBAL_CTRL MSR) before changes to the configuration (e.g. what event is specified in 
IA32_PERFEVTSELx or whether PEBS is enabled for that counter via IA32_PEBS_ENABLE) or counter value (MSR 
write to IA32_PMCx and IA32_A_PMCx). 


18.5.3.1.3 | Enhancements to |A32_PERF_GLOBAL_STATUS.OvfDSBuffer[62] 


In addition to IA32_PERF_GLOBAL_STATUS.OvfDSBuffer[62] being set when PEBS_Index reaches the 
PEBS_Interrupt_Theshold, the bit is also set when PEBS_Index is out of bounds. That is, the bit will be set when 
PEBS_Index < PEBS_Buffer_Base or PEBS_Index > PEBS_Absolute_Maximum. Note that when an out of bound 
condition is encountered, the overflow bits in IA32_PERF_GLOBAL_STATUS will be cleared according to Applicable 
Counters, however the IA32_PMCx values will not be reloaded with the Reset values stored in the DS_AREA. 


18.5.3.2 Offcore Response Event 


Event number 0B7H support offcore response monitoring using an associated configuration MSR, 
MSR_OFFCORE_RSPO (address 1A6H) in conjunction with umask value 01H or MSR_OFFCORE_RSP1 (address 
1A7H) in conjunction with umask value 02H. Table 18-56 lists the event code, mask value and additional off-core 
configuration MSR that must be programmed to count off-core response events using IA32_PMCx. 


The Goldmont microarchitecture provides unique pairs of MSR_OFFCORE_RSPx registers per core. 
The layout of MSR_OFFCORE_RSPO and MSR_OFFCORE_RSP1 are organized as follows: 
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° Bits 15:0 specifies the request type of a transaction request to the uncore. This is described in Table 18-63. 
¢ Bits 30:16 specifies common supplier information or an L2 Hit, and is described in Table 18-58. 


° If L2 misses, then Bits 37:31 can be used to specify snoop response information and is described in 
Table 18-64. 


° For outstanding requests, bit 38 can enable measurement of average latency of specific type of offcore 
transaction requests using two programmable counter simultaneously; see Section 18.5.2.3 for details. 


Table 18-63. MSR_OFFCORE_RSPx Request_Type Field Definition 


Bit Name Offset | Description 

DEMAND_DATA_RD 0 Counts cacheline read requests due to demand reads (excludes prefetches). 

DEMAND_RFO 1 Counts cacheline read for ownership (RFO) requests due to demand writes (excludes 
prefetches). 

DEMAND_CODE_RD 2 Counts demand instruction cacheline and I-side prefetch requests that miss the 


instruction cache. 


COREWB 3 Counts writeback transactions caused by L1 or L2 cache evictions. 

PF_L2_DATA_RD 4 Counts data cacheline reads generated by hardware L2 cache prefetcher. 
PF_L2_RFO 5 Counts reads for ownership (RFO) requests generated by L2 prefetcher. 

Reserved 6 Reserved. 

PARTIAL_READS 7 Counts demand data partial reads, including data in uncacheable (UC) or uncacheable 


(WC) write combining memory types. 


PARTIAL_WRITES 8 Counts partial writes, including uncacheable (UC), write through (WT) and write 
protected (WP) memory type writes. 


UC_CODE_READS 9 Counts code reads in uncacheable (UC) memory region. 

BUS_LOCKS 10 Counts bus lock and split lock requests. 

FULL_STREAMING_STORES 11 Counts full cacheline writes due to streaming stores. 

Sw_PREFETCH 12 Counts cacheline requests due to software prefetch instructions. 
PF_L1_DATA_RD 13 Counts data cacheline reads generated by hardware L1 data cache prefetcher. 
PARTIAL_STREAMING_STORES 14 Counts partial cacheline writes due to streaming stores. 

ANY_REQUEST 15 Counts requests to the uncore subsystem. 


To properly program this extra register, software must set at least one request type bit (Table 18-57) and a valid 
response type pattern (either Table 18-58 or Table 18-64). Otherwise, the event count reported will be zero. It is 
permissible and useful to set multiple request and response type bits in order to obtain various classes of off-core 
response events. Although MSR_OFFCORE_RSPx allow an agent software to program numerous combinations that 
meet the above guideline, not all combinations produce meaningful data. 
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Table 18-64. MSR_OFFCORE_RSPx For L2 Miss and Outstanding Requests 


Subtype Bit Name Offset | Description 
L2_MISS Reserved 32:31 Reserved 
(Snoop Info) L2_MISS.SNOOP_MISS_O | 33 A true miss to this module, for which a snoop request missed the other module or 
R_NO_SNOOP_NEEDED no snoop was performed/needed. 
L2_MISS.HIT_OTHER_CO | 34 A snoop hit in the other processor module, but no data forwarding is required. 
RE_NO_FWD 
Reserved 35 Reserved 
L2_MISS.HITM_OTHER_C | 36 Counts the number of snoops hit in the other module or other core’s L1 where 
ORE modified copies were found. 
L2_MISS.NON_DRAM 37 Target was a non-DRAM system address. This includes MMIO transactions. 
Outstanding | OUTSTANDING 38 Counts weighted cycles of outstanding offcore requests of the request type 
requests! specified in bits 15:0, from the time the XQ receives the request and any 
response is received. Bits 37:16 must be set to O. This bit is only available in 
MSR_OFFCORE_RESPO. 


NOTES: 
1. See Section 18.5.2.3, “Average Offcore Request Latency Measurement” for details on how to use this bit to extract average latency. 


To specify a complete offcore response filter, software must properly program bits in the request and response type 
fields. A valid request type must have at least one bit set in the non-reserved bits of 15:0. A valid response type 
must be a non-zero value of the following expression: 


Any_Response Bit | L2 Hit | ‘OR’ of Snoop Info Bits | Outstanding Bit 


18.5.3.3 Average Offcore Request Latency Measurement 


In Goldmont microarchitecture, measurement of average latency of offcore transaction requests is the same as 
described in Section 18.5.2.3. 


18.5.4 Performance Monitoring for Goldmont Plus Microarchitecture 


Intel Atom processors based on the Goldmont Plus microarchitecture report architectural performance monitoring 
versionID = 4 and support non-architectural monitoring capabilities described in this section. 


Architectural performance monitoring version 4 capabilities are described in Section 18.2.4. 


Goldmont Plus performance monitoring capabilities are similar to Goldmont capabilities. The differences are in 
specific events and in which counters support PEBS. Goldmont Plus introduces the ability for fixed performance 
monitoring counters to generate PEBS records. 


Goldmont Plus will set the AnyThread deprecation CPUID bit (CPUID.OAH:EDX[15]) to indicate that the Any-Thread 
bits in IA32_PERFEVTSELx and IA32_FIXED_CTR_CTRL have no effect. 


The core PMU's capability is similar to that of the Goldmont microarchitecture described in Section 18.6.3, with 
some differences and enhancements summarized in Table 18-65. 
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Table 18-65. Core PMU Comparison Between the Goldmont Plus and Goldmont Microarchitectures 


Box Goldmont Plus Microarchitecture | Goldmont Microarchitecture Comment 
# of Fixed counters per core | 3 3 Use CPUID to determine # 
of counters. See Section 
18.2.1. 
# of general-purpose 4 4 Use CPUID to determine # 
counters per core of counters. See Section 
18.2.1. 
Counter width (R,W) R:48, W: 32/48 R:48, W: 32/48 No change. 
Architectural Performance 4 4 No change. 
Monitoring version ID 
Processor Event Based All General-Purpose and Fixed General-Purpose Counter O only. Goldmont Plus supports 
Sampling (PEBS) Events counters. Each General-Purpose Supports all events (precise and PEBS on all counters. 
counter supports all events (precise | non-precise). Precise events are 
and non-precise). listed in Table 18-61. 
PEBS record format 0011b 0011b No change. 
encoding 


18.5.4.1 Extended PEBS 


The PEBS facility in Goldmont Plus microarchitecture provides a number of enhancements relative to PEBS in 
processors from previous generations. Enhancement of PEBS facility with the Extended PEBS feature are de- 
scribed in detail in section 18.9. 


18.5.5 Performance Monitoring for Tremont Microarchitecture 


Intel Atom processors based on the Tremont microarchitecture report architectural performance monitoring 
versionID = 5 and support non-architectural monitoring capabilities described in this section. 


Architectural performance monitoring version 5 capabilities are described in Section 18.2.5. 


Tremont performance monitoring capabilities are similar to Goldmont Plus capabilities, with the following exten- 
sions: 


° Support for Adaptive PEBS. 

* Support for PEBS output to Intel® Processor Trace. 

° Precise Distribution support on Fixed Counter0. 

* Compatibility enhancements to off-core response MSRs, MSR_OFFCORE_RSPx. 


The differences and enhancements between Tremont microarchitecture and Goldmont Plus microarchitecture are 
summarized in Table 18-66. 


18-88 Vol. 3B 


PERFORMANCE MONITORING 


Table 18-66. Core PMU Comparison Between the Tremont and Goldmont Plus Microarchitectures 


Box 


Tremont Microarchitecture 


Goldmont Plus Microarchitecture 


Comment 


# of fixed counters per core 


3 


3 


Use CPUID to determine # 
of counters. See Section 
18.2.1. 


Trace 


# of general-purpose 4 4 Use CPUID to determine # 

counters per core of counters. See Section 
18.2.1. 

Counter width (R,W) R:48, W: 32/48 R:48, W: 32/48 No change. See Section 
18.2.2. 

Architectural Performance 5 4 

Monitoring version ID 

PEBS record format 0100b 0011b See Section 18.6.2.4.2. 

encoding 

Reduce skid PEBS IA32_PMCO and IA32_FIXED_CTRO | IA32_PMCO only 

Extended PEBS Yes Yes See Section 18.5.4.1. 

Adaptive PEBS Yes No See Section 18.9.2. 

PEBS output DS Save Area or Intel® Processor DS Save Area only See Section 18.5.5.2.1. 


PEBS record layout 


See Section 18.9.2.3 for output to 
DS, Section 18.5.5.2.2 for output to 
Intel PT. 


Table 18-62; enhanced fields at 
offsets 9OH- 98H; and TSC record 
field at COH. 


Off-core Response Event 


MSR 1A6H and 1A7H, each core 
has its own register, extended 
request and response types. 


MSR 1A6H and 1A7H, each core has 
its own register. 


18.5.5.1 


Adaptive PEBS 


The PEBS record format and configuration interface has changed versus Goldmont Plus, as the Tremont microar- 
chitecture includes support for the configurable Adaptive PEBS records; see Section 18.9.2. 


18.5.5.2 PEBS output to Intel® Processor Trace 


Intel Atom processors based on the Tremont microarchitecture introduce the following Precise Event-Based 
Sampling (PEBS) extensions: 


* Amechanism to direct PEBS output into the Intel® Processor Trace (Intel® PT) output stream. In this scenario, 
the PEBS record is written in packetized form, in order to co-exist with other Intel PT trace data. 


¢ New Performance Monitoring counter reload MSRs, which are used by PEBS in place of the counter reload 
values stored in the DS Management area when PEBS output is directed into the Intel PT output stream. 


Processors that indicate support for Intel PT by setting CPUID.07H.0.EBX[25]=1, and set the new 
IA32_PERF_CAPABILITIES.PEBS_OUTPUT_PT_AVAIL[16] bit, support these extensions. 


18.5.5.2.1 


PEBS Configuration 


PEBS output to Intel Processor Trace includes support for two new fields in IA32_PEBS_ENABLE. 
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Table 18-67. New Fields in |A32_PEBS_ENABLE 


Field Description 
PMI_AFTER_EACH_RECORD[60] Pend a PerfMon Interrupt (PMI) after each PEBS event. 
PEBS_OUTPUT[62:61] Specifies PEBS output destination. Encodings: 


OOB: DS Save Area. Matches legacy PEBS behavior, output location defined by [A32_DS_AREA. 
01B: Intel PT trace output. 

10B: Reserved. 

11B: Reserved. 


When PEBS_OUTPUT is set to 01B, the DS Management Area is not used and need not be configured. Instead, the 
output mechanism is configured through IA32_RTIT_CTL and other Intel PT MSRs, while counter reload values are 
configured in the MSR_RELOAD_PMCx MSRs. Details on configuring Intel PT can be found in Section 31.2.6. 


63 626160 n 32 31 ‘i 10 


PEBS OUTPUT (R/W) 


PMI_AFTER_EACH_RECORD (R/W) ——~ 


PEBS EN_FIXEDn(R/W) 


PEBS EN_FIXED1 (R/W) 


PEBS_EN_FIXEDO (R/W) 


PEBS EN_PMCm (RW) 


PEBS EN _PMC1 (RM) 


PEBS EN_PMCO (RM) 


Reserved RESET Value — 00000000 _ 00000000 H 


Figure 18-40. IA32_PEBS_ENABLE MSR with PEBS Output to Intel® Processor Trace 


18.5.5.2.2 PEBS Record Format in Intel® Processor Trace 


The format of the PEBS record changes when output to Intel PT, as the PEBS state is packetized. Each PEBS 
grouping is emitted as a Block Begin (BBP) and following Block Item (BIP) packets. A PEBS grouping ends when 
either a new PEBS grouping begins (indicated by a BBP packet) or a Block End (BEP) packet is encountered. See 
Section 31.4.1.1 for details of these Intel PT packets. 


Because the packet headers describe the state held in the packet payload, PEBS state ordering is not fixed. PEBS 
state groupings may be emitted in any order, and the PEBS state elements within those groupings may be emitted 
in any order. Further, there is no packet that provides indication of “Record Format” or “Record Size”. 


If Intel PT tracing is not enabled (IA32_RTIT_STATUS.TriggerEn=0), any PEBS records triggered will be dropped. 
PEBS packets do not depend on ContextEn or FilterEn in IA32_RTIT_STATUS, any filtering of PEBS must be enabled 
from within the PerfMon configuration. Counter reload will occur in all scenarios where PEBS is triggered, regardless 
of TriggerEn. 


18-90 Vol. 3B 


PERFORMANCE MONITORING 


The PEBS threshold mechanism for generating PerfMon Interrupts (PMIs) is not available in this mode. However, 
there exist other means to generate PMIs based on PEBS output. When the Intel PT ToPA output mechanism is 
chosen, a PMI can optionally be pended when a ToPA region is filled; see Section 31.2.6.2 for details. Further, soft- 
ware can opt to generate a PMI on each PEBS record by setting the new 
IA32_PEBS_ENABLE.PMI_AFTER_EACH_RECORD[60] bit. 


The IA32_PERF_GLOBAL_STATUS.OvfDSBuffer bit will not be set in this mode. 


18.5.5.2.3 PEBS Counter Reload 


When PEBS output is directed into Intel PT (IA32_PEBS_ENABLE.PEBS_OUTPUT = 01B), new MSR_RELOAD_PMCx 
MSRs are used by the PEBS routine to reload PerfMon counters. The value from the associated reload MSR will be 
loaded to the appropriate counter on each PEBS event. 


18.5.5.3. Precise Distribution Support on Fixed Counter 0 


The Tremont microarchitecture supports the PDIR (Precise Distribution of Retired Instructions) facility, as described 
in Section 18.3.4.4.4, on Fixed Counter 0. Fixed Counter 0 counts the INST_RETIRED.ALL event. PEBS skid for 
Fixed Counter 0 will be precisely one instruction. 


This is in addition to the reduced skid PEBS behavior on IA32_PMCO; see Section 18.5.3.1.2. 


18.5.5.4 Compatibility Enhancements to Offcore Response MSRs 
The Off-core Response facility is similar to that described in Section 18.5.3.2. 


The layout of MSR_OFFCORE_RSPO and MSR_OFFCORE_RSP1 are organized as shown below. RequestType bits are 
defined in Table 18-68, ResponseType bits in Table 18-69, and SnoopInfo bits in Table 18-70. 


Table 18-68. MSR_OFFCORE_RSPx Request Type Definition 


Bit Name Offset Description 

DEMAND_DATA_RD 0 Counts demand data reads. 

DEMAND_RFO 1 Counts all demand reads for ownership (RFO) requests and software based 
prefetches for exclusive ownership (prefetchw). 

DEMAND_CODE_RD 2 Counts demand instruction fetches and L1 instruction cache prefetches. 

COREWB_M 3 Counts modified write backs from L1 and L2. 

HWPF_L2_ DATA_RD 4 Counts prefetch (that bring data to L2) data reads. 

HWPF_L2_RFO 5 Counts all prefetch (that bring data to L2) RFOs. 

HWPF_L2_CODE_RD 6 Counts all prefetch (that bring data to L2 only) code reads. 

Reserved 9:7 Reserved. 

HWPF_L1D_AND_SWPF 10 Counts L1 data cache hardware prefetch requests, read for ownership prefetch 
requests and software prefetch requests (except prefetchw). 

STREAMING_WR 11 Counts all streaming stores. 

COREWB_NONM 12 Counts non-modified write backs from L2. 

Reserved 14:13 Reserved. 

OTHER 15 Counts miscellaneous requests, such as I/O accesses that have any response type. 

UC_RD 44 Counts uncached memory reads (PRd, UCRdF). 

UC_WR 45 Counts uncached memory writes (Wil). 

PARTIAL_STREAMING_WR 46 Counts partial (less than 64 byte) streaming stores (WCIL). 

FULL_STREAMING_WR 47 Counts full, 64 byte streaming stores (WCILF). 


Vol. 3B 18-91 


PERFORMANCE MONITORING 


Table 18-68. MSR_OFFCORE_RSPx Request Type Definition 


Bit Name Offset Description 
LIWB_M 48 Counts modified WriteBacks from L1 that miss the L2. 
L2WB_M 49 Counts modified WriteBacks from L2. 
Table 18-69. MSR_OFFCORE_RSPx Response Type Definition 
Bit Name Offset Description 
ANY_RESPONSE 16 Catch all value for any response types. 
L3_HIT_M 18 LLC/L3 Hit - M-state. 
L3_HIT_E 19 LLC/L3 Hit - €-state. 
L3_HIT_S 20 LLC/L3 Hit - S-state. 
L3_HIT_F 21 LLC/L3 Hit - I-state. 
LOCAL_DRAM 26 LLC/L3 Miss, DRAM Hit. 
OUTSTANDING 63 Average latency of outstanding requests with the other counter counting number 
of occurrences; can also can be used to count occupancy. 
Table 18-70. MSR_OFFCORE_RSPx Snoop Info Definition 
Bit Name Offset Description 
SNOOP_NONE 311 None of the cores were snooped. 
* LLC miss and Dram data returned directly to the core. 
SNOOP_NOT_NEEDED 32 No snoop needed to satisfy the request. 
* LLC hit and CV bit(s) (core valid) was not set. 
= LLC miss and Dram data returned directly to the core. 
SNOOP_MISS 33 A snoop was sent but missed. 
* LLC hit and CV bit(s) was set but snoop missed (silent data drop in core), data 
returned from LLC. 
* LLC miss and Dram data returned directly to the core. 
SNOOP_HIT_NO_FWD 34 A snoop was sent but no data forward. 
* LLC hit and CV bit(s) was set but no data forward from the core, data returned 
from LLC. 
* LLC miss and Dram data returned directly to the core. 
SNOOP_HIT_WITH_FWD 35 A snoop was sent and non-modified data was forward. 
* LLC hit and CV bit(s) was set, non-modified data was forward from core. 
SNOOP_HITM 36 A snoop was sent and modified data was forward. 
* LLC hit € or M and the CV bit(s) was set, modified data was forward from core. 
NON_DRAM_BIT 37 Target was non-DRAM system address, MMIO access. 
* LLC miss and Non-Dram data returned. 
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The Off-core Response capability behaves as follows: 


° To specify a complete offcore response filter, software must properly program at least one RequestType and one 
ResponseType. A valid request type must have at least one bit set in the non-reserved bits of 15:0 or 49:44.A 
valid response type must be a non-zero value of one the following expressions: 


¢ Read requests: 
Any_Response Bit | (‘(OR’ of Supplier Info Bits) ‘AND’ ( ‘OR’ of Snoop Info Bits) | Outstanding Bit 
¢ Write requests: 
Any_Response Bit | (OR’ of Supplier Info Bits) | Outstanding Bit 
¢ When the ANY_RESPONSE bit in the ResponseType is set, all other response type bits will be ignored. 
¢ True Demand Cacheable Loads include neither L1 Prefetches nor Software Prefetches. 


e Bits 15:0 and Bits 49:44 specifies the request type of a transaction request to the uncore. This is described in 
Table 18-68. 


° Bits 30:16 specifies common supplier information. 


¢ “Outstanding Requests” (bit 63) is only available on MSR_OFFCORE_RSPO; a #GP fault will occur if software 
attempts to write a 1 to this bit in MSR_OFFCORE_RSP1. It is mutually exclusive with any ResponseType. 
Software must guarantee that all other ResponseType bits are set to 0 when the “Outstanding Requests” bit is 
set. 


¢ “Outstanding Requests” bit 63 can enable measurement of the average latency of a specific type of off-core 
transaction; two programmable counters must be used simultaneously and the RequestType programming for 
MSR_OFFCORE_RSPO and MSR_OFFCORE_RSP1 must be the same when using this Average Latency feature. 
See Section 18.5.2.3 for further details. 


18.6 PERFORMANCE MONITORING (LEGACY INTEL PROCESSORS) 


18.6.1 | Performance Monitoring (Intel® Core™ Solo and Intel® Core™ Duo Processors) 


In Intel Core Solo and Intel Core Duo processors, non-architectural performance monitoring events are 
programmed using the same facilities (see Figure 18-1) used for architectural performance events. 


Non-architectural performance events use event select values that are model-specific. Event mask (Umask) values 
are also specific to event logic units. Some microarchitectural conditions detectable by a Umask value may have 
specificity related to processor topology (see Section 8.6, “Detecting Hardware Multi-Threading Support and 
Topology,” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). As a result, the unit 
mask field (for example, IA32_PERFEVTSELx[bits 15:8]) may contain sub-fields that specify topology information 
of processor cores. 


The sub-field layout within the Umask field may support two-bit encoding that qualifies the relationship between a 
microarchitectural condition and the originating core. This data is shown in Table 18-71. The two-bit encoding for 
core-specificity is only supported for a subset of Umask values (see: https://perfmon-events.intel.com/) and for 

Intel Core Duo processors. Such events are referred to as core-specific events. 


Table 18-71. Core Specificity Encoding within a Non-Architectural Umask 


IA32_PERFEVTSELx MSRs 

Bit 15:14 Encoding Description 
11B All cores 
10B Reserved 
01B This core 
00B Reserved 


Vol. 3B 18-93 


PERFORMANCE MONITORING 


Some microarchitectural conditions allow detection specificity only at the boundary of physical processors. Some 
bus events belong to this category, providing specificity between the originating physical processor (a bus agent) 
versus other agents on the bus. Sub-field encoding for agent specificity is shown in Table 18-72. 


Table 18-72. Agent Specificity Encoding within a Non-Architectural Umask 


IA32_PERFEVTSELx MSRs 

Bit 13 Encoding Description 

6) This agent 

1 Include all agents 


Some microarchitectural conditions are detectable only from the originating core. In such cases, unit mask does 
not support core-specificity or agent-specificity encodings. These are referred to as core-only conditions. 


Some microarchitectural conditions allow detection specificity that includes or excludes the action of hardware 
prefetches. A two-bit encoding may be supported to qualify hardware prefetch actions. Typically, this applies only 
to some L2 or bus events. The sub-field encoding for hardware prefetch qualification is shown in Table 18-73. 


Table 18-73. HW Prefetch Qualification Encoding within a Non-Architectural Umask 


IA32_PERFEVTSELx MSRs 

Bit 13:12 Encoding Description 

11B All inclusive 

10B Reserved 

01B Hardware prefetch only 
OOB Exclude hardware prefetch 


Some performance events may (a) support none of the three event-specific qualification encodings (b) may 
support core-specificity and agent specificity simultaneously (c) or may support core-specificity and hardware 
prefetch qualification simultaneously. Agent-specificity and hardware prefetch qualification are mutually exclusive. 


In addition, some L2 events permit qualifications that distinguish cache coherent states. The sub-field definition for 
cache coherency state qualification is shown in Table 18-74. If no bits in the MESI qualification sub-field are set for 
an event that requires setting MESI qualification bits, the event count will not increment. 


Table 18-74. MESI Qualification Definitions within a Non-Architectural Umask 


IA32_PERFEVTSELx MSRs 

Bit Position 11:8 Description 

Bit 11 Counts modified state 
Bit 10 Counts exclusive state 
Bit 9 Counts shared state 
Bit 8 Counts Invalid state 


18.6.2 Performance Monitoring (Processors Based on Intel® Core™” Microarchitecture) 


In addition to architectural performance monitoring, processors based on the Intel Core microarchitecture support 
non-architectural performance monitoring events. 


Architectural performance events can be collected using general-purpose performance counters. Non-architectural 
performance events can be collected using general-purpose performance counters (coupled with two 
IA32_PERFEVTSELx MSRs for detailed event configurations), or fixed-function performance counters (see Section 
18.6.2.1). IA32_PERFEVTSELx MSRs are architectural; their layout is shown in Figure 18-1. Starting with Intel 
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Core 2 processor T 7700, fixed-function performance counters and associated counter control and status MSR 
becomes part of architectural performance monitoring version 2 facilities (see also Section 18.2.2). 


Non-architectural performance events in processors based on Intel Core microarchitecture use event select values 
that are model-specific. Valid event mask (Umask) bits can be found at: https://perfmon-events.intel.com/. The 
UMASK field may contain sub-fields identical to those listed in Table 18-71, Table 18-72, Table 18-73, and 

Table 18-74. One or more of these sub-fields may apply to specific events on an event-by-event basis. 


In addition, the UMASK filed may also contain a sub-field that allows detection specificity related to snoop 
responses. Bits of the snoop response qualification sub-field are defined in Table 18-75. 


Table 18-75. Bus Snoop Qualification Definitions within a Non-Architectural Umask 


IA32_PERFEVTSELx MSRs 

Bit Position 11:8 Description 

Bit 11 HITM response 
Bit 10 Reserved 

Bit 9 HIT response 
Bit 8 CLEAN response 


There are also non-architectural events that support qualification of different types of snoop operation. The corre- 
sponding bit field for snoop type qualification are listed in Table 18-76. 


Table 18-76. Snoop Type Qualification Definitions within a Non-Architectural Umask 


IA32_PERFEVTSELx MSRs 

Bit Position 9:8 Description 
Bit 9 CMP2l snoops 
Bit 8 CMP2S snoops 


No more than one sub-field of MESI, snoop response, and snoop type qualification sub-fields can be supported ina 
performance event. 


NOTE 


Software must write known values to the performance counters prior to enabling the counters. The 
content of general-purpose counters and fixed-function counters are undefined after INIT or 
RESET. 


18.6.2.1 Fixed-function Performance Counters 


Processors based on Intel Core microarchitecture provide three fixed-function performance counters. Bits beyond 
the width of the fixed counter are reserved and must be written as zeros. Model-specific fixed-function perfor- 
mance counters on processors that support Architectural Perfmon version 1 are 40 bits wide. 


Each of the fixed-function counter is dedicated to count a pre-defined performance monitoring events. See Table 
18-2 for details of the PMC addresses and what these events count. 


Programming the fixed-function performance counters does not involve any of the IA32_PERFEVTSELx MSRs, and 
does not require specifying any event masks. Instead, the MSR IA32_FIXED_CTR_CTRL provides multiple sets of 
4-bit fields; each 4-bit field controls the operation of a fixed-function performance counter (PMC). See Figures 
18-41. Two sub-fields are defined for each control. See Figure 18-41; bit fields are: 


¢ Enable field (low 2 bits in each 4-bit control) — When bit 0 is set, performance counting is enabled in the 
corresponding fixed-function performance counter to increment when the target condition associated with the 
architecture performance event occurs at ring 0. 
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When bit 1 is set, performance counting is enabled in the corresponding fixed-function performance counter to 
increment when the target condition associated with the architecture performance event occurs at ring greater 
than 0. 

Writing 0 to both bits stops the performance counter. Writing 11B causes the counter to increment irrespective 
of privilege levels. 


63 1211 987 543210 


P P 
M M 
I I 


Cntr2 — Controls for |A32_FIXED_CTR2 
Cntr1 — Controls for |A832_FIXED_CTR1 


PMI — Enable PMI on overflow 
CntrO0 — Controls for |A32_FIXED_CTRO 
ENABLE — 0: disable; 1: OS; 2: User; 3: All ring levels 


Reserved 


SDM30265 


Figure 18-41. Layout of IA32_FIXED_CTR_CTRL MSR 


° PMI field (fourth bit in each 4-bit control) — When set, the logical processor generates an exception 
through its local APIC on overflow condition of the respective fixed-function counter. 


18.6.2.2 Global Counter Control Facilities 


Processors based on Intel Core microarchitecture provides simplified performance counter control that simplifies 

the most frequent operations in programming performance events, i.e. enabling/disabling event counting and 

checking the status of counter overflows. This is done by the following three MSRs: 

¢ MSR_PERF_GLOBAL_CTRL enables/disables event counting for all or any combination of fixed-function PMCs 
(IA32_FIXED_CTRx) or general-purpose PMCs via a single WRMSR. 

¢ MSR_PERF_GLOBAL_STATUS allows software to query counter overflow conditions on any combination of 
fixed-function PMCs (IA32_FIXED_CTRx) or general-purpose PMCs via a single RDMSR. 


¢ MSR_PERF_GLOBAL_OVF_CTRL allows software to clear counter overflow conditions on any combination of 
fixed-function PMCs (IA32_FIXED_CTRx) or general-purpose PMCs via a single WRMSR. 


MSR_PERF_GLOBAL_CTRL MSR provides single-bit controls to enable counting in each performance counter (see 
Figure 18-42). Each enable bit in MSR_PERF_GLOBAL_CTRL is AND’ed with the enable bits for all privilege levels in 
the respective IA32_PERFEVTSELx or IA32_FIXED_CTR_CTRL MSRs to start/stop the counting of respective coun- 
ters. Counting is enabled if the AND’ed results is true; counting is disabled when the result is false. 
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63 35 34 33 32 31 210 


FIXED_CTR2 enable 
FIXED_CTR1 enable 
FIXED_CTRO enable 
PMC1 enable 
PMCO enable 


Reserved 


Figure 18-42. Layout of MSR_PERF_GLOBAL_CTRL MSR 


MSR_PERF_GLOBAL_STATUS MSR provides single-bit status used by software to query the overflow condition of 
each performance counter. MSR_PERF_GLOBAL_STATUS[bit 62] indicates overflow conditions of the DS area data 
buffer. MSR_PERF_GLOBAL_STATUS[bit 63] provides a CondChgd bit to indicate changes to the state of perfor- 
mance monitoring hardware (see Figure 18-43). A value of 1 in bits 34:32, 1, 0 indicates an overflow condition has 
occurred in the associated counter. 


6362 35 3433 3231 210 


CondChgd 
OvfBuffer 
FIXED_CTR2 Overflow 
FIXED_CTR1 Overflow 
FIXED_CTRO Overflow 
PMC1 Overflow 
PMCO Overflow 


Reserved 


Figure 18-43. Layout of MSR_PERF_GLOBAL_STATUS MSR 


When a performance counter is configured for PEBS, an overflow condition in the counter will arm PEBS. On the 
subsequent event following overflow, the processor will generate a PEBS event. On a PEBS event, the processor will 
perform bounds checks based on the parameters defined in the DS Save Area (see Section 17.4.9). Upon 
successful bounds checks, the processor will store the data record in the defined buffer area, clear the counter 
overflow status, and reload the counter. If the bounds checks fail, the PEBS will be skipped entirely. In the event 
that the PEBS buffer fills up, the processor will set the OvfBuffer bit in MSR_PERF_GLOBAL_STATUS. 
MSR_PERF_GLOBAL_OVF_CTL MSR allows software to clear overflow the indicators for general-purpose or fixed- 
function counters via a single WRMSR (see Figure 18-44). Clear overflow indications when: 

¢ Setting up new values in the event select and/or UMASK field for counting or interrupt-based event sampling. 
¢ Reloading counter values to continue collecting next sample. 


* Disabling event counting or interrupt-based event sampling. 
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6362 35 34.33 32 31 210 


dieacnaa 
ClrOvfBuffer 

FIXED_CTR2 ClrOverflow 
FIXED_CTR1 ClrOverflow 
FIXED_CTRO ClrOverflow 
PMC1 ClrOverflow 
PMCO ClrOverflow 


Reserved 


Figure 18-44. Layout of MSR_PERF_GLOBAL_OVF_CTRL MSR 


18.6.2.3 At-Retirement Events 


Many non-architectural performance events are impacted by the speculative nature of out-of-order execution. A 
subset of non-architectural performance events on processors based on Intel Core microarchitecture are enhanced 
with a tagging mechanism (similar to that found in Intel NetBurst® microarchitecture) that exclude contributions 
that arise from speculative execution. The at-retirement events available in processors based on Intel Core micro- 
architecture does not require special MSR programming control (see Section 18.6.3.6, “At-Retirement Counting”), 
but is limited to IA32_PMCO. See Table 18-77 for a list of events available to processors based on Intel Core micro- 
architecture. 


Table 18-77. At-Retirement Performance Events for Intel Core Microarchitecture 


Event Name UMask Event Select 
ITLB_MISS_RETIRED OOH C9H 
MEM_LOAD_RETIRED.L1D_MISS 01H CBH 
MEM_LOAD_RETIRED.L1D_LINE_MISS 02H CBH 
MEM_LOAD_RETIRED.L2_MISS 04H CBH 
MEM_LOAD_RETIRED.L2_LINE_MISS 08H CBH 
MEM_LOAD_RETIRED.DTLB_MISS 10H CBH 


18.6.2.4 Processor Event Based Sampling (PEBS) 


Processors based on Intel Core microarchitecture also support processor event based sampling (PEBS). This 
feature was introduced by processors based on Intel NetBurst microarchitecture. 


PEBS uses a debug store mechanism and a performance monitoring interrupt to store a set of architectural state 
information for the processor. The information provides architectural state of the instruction executed after the 
instruction that caused the event (See Section 18.6.2.4.2 and Section 17.4.9). 


In cases where the same instruction causes BTS and PEBS to be activated, PEBS is processed before BTS are 
processed. The PMI request is held until the processor completes processing of PEBS and BTS. 


For processors based on Intel Core microarchitecture, precise events that can be used with PEBS are listed in 
Table 18-78. The procedure for detecting availability of PEBS is the same as described in Section 18.6.3.8.1. 
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Table 18-78. PEBS Performance Events for Intel Core Microarchitecture 


Event Name UMask Event Select 
INSTR_RETIRED.ANY_P OOH COH 
X87_OPS_RETIRED.ANY FEH C1H 
BR_INST_RETIRED.MISPRED OOH C5H 
SIMD_INST_RETIRED.ANY 1FH C7H 
MEM_LOAD_RETIRED.L1D_MISS 01H CBH 
MEM_LOAD_RETIRED.L1D_LINE_MISS 02H CBH 
MEM_LOAD_RETIRED.L2_MISS 04H CBH 
MEM_LOAD_RETIRED.L2_LINE_MISS 08H CBH 
MEM_LOAD_RETIRED.DTLB_MISS 10H CBH 


18.6.2.4.1 Setting up the PEBS Buffer 


For processors based on Intel Core microarchitecture, PEBS is available using IA32_PMCO only. Use the following 
procedure to set up the processor and IA32_PMCO counter for PEBS: 


1. Setup the precise event buffering facilities. Place values in the precise event buffer base, precise event index, 
precise event absolute maximum, precise event interrupt threshold, and precise event counter reset fields of 
the DS buffer management area. In processors based on Intel Core microarchitecture, PEBS records consist of 
64-bit address entries. See Figure 17-8 to set up the precise event records buffer in memory. 


Enable PEBS. Set the Enable PEBS on PMCO flag (bit 0) in IA32_PEBS_ENABLE MSR. 
Set up the IA32_PMCO performance counter and IA32_PERFEVTSELO for an event listed in Table 18-78. 


18.6.2.4.2 PEBS Record Format 


The PEBS record format may be extended across different processor implementations. The 
IA32_PERF_CAPABILITES MSR defines a mechanism for software to handle the evolution of PEBS record format in 
processors that support architectural performance monitoring with version id equals 2 or higher. The bit fields of 
IA32_PERF_CAPABILITES are defined in Table 2-2 of Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 4. The relevant bit fields that governs PEBS are: 


¢ PEBSTrap [bit 6]: When set, PEBS recording is trap-like. After the PEBS-enabled counter has overflowed, PEBS 
record is recorded for the next PEBS-able event at the completion of the sampled instruction causing the PEBS 
event. When clear, PEBS recording is fault-like. The PEBS record is recorded before the sampled instruction 
causing the PEBS event. 


¢ PEBSSaveArchRegs [bit 7]: When set, PEBS will save architectural register and state information according to 
the encoded value of the PEBSRecordFormat field. When clear, only the return instruction pointer and flags are 
recorded. On processors based on Intel Core microarchitecture, this bit is always 1. 


¢ PEBSRecordFormat [bits 11:8]: Valid encodings are: 


— OOOOB: Only general-purpose registers, instruction pointer and RFLAGS registers are saved in each PEBS 
record (See Section 18.6.3.8). 


— 0001B: PEBS record includes additional information of IA32_PERF_GLOBAL_STATUS and load latency data. 
(See Section 18.3.1.1.1). 


— 0010B: PEBS record includes additional information of IA32_PERF_GLOBAL_STATUS, load latency data, 
and TSX tuning information. (See Section 18.3.6.2). 


— 0011B: PEBS record includes additional information of load latency data, TSX tuning information, TSC data, 
and the applicable counter field replaces IA32_PERF_GLOBAL_STATUS at offset 90H. (See Section 
18.3.8.1.1). 


— 0100B: PEBS record contents are defined by elections in MSR_PEBS_DATA_CFG. (See Section 18.9.2.3). 
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18.6.2.4.3 Writing a PEBS Interrupt Service Routine 


The PEBS facilities share the same interrupt vector and interrupt service routine (called the DS ISR) with the Inter- 
rupt-based event sampling and BTS facilities. To handle PEBS interrupts, PEBS handler code must be included in 
the DS ISR. See Section 17.4.9.1, “64 Bit Format of the DS Save Area,” for guidelines when writing the DS ISR. 


The service routine can query MSR_PERF_GLOBAL_STATUS to determine which counter(s) caused of overflow 
condition. The service routine should clear overflow indicator by writing to MSR_PERF_GLOBAL_OVF_CTL. 


A comparison of the sequence of requirements to program PEBS for processors based on Intel Core and Intel 
NetBurst microarchitectures is listed in Table 18-79. 


Table 18-79. Requirements to Program PEBS 


For Processors based on Intel Core 
microarchitecture 


For Processors based on Intel NetBurst 
microarchitecture 


Verify PEBS support of 
processor/OS. 


* IA32_MISC_ENABLE.EMON_AVAILABE (bit 7) is set. 


* IA32_MISC_ENABLE.PEBS_UNAVAILABE (bit 12) is clear. 


Ensure counters are in disabled. 


On initial set up or changing event configurations, 
write MSR_PERF_GLOBAL_CTRL MSR (38FH) with 0. 


On subsequent entries: 


* Clear all counters if “Counter Freeze on PMI" is not 
enabled. 

« If 1A32_DebugCTL.Freeze is enabled, counters are 
automatically disabled. 

Counters MUST be stopped before writing. ! 


Optional 


Disable PEBS. 


Clear ENABLE PMCO bit in IA32_PEBS_ENABLE MSR 
(3F1H). 


Optional 


Check overflow conditions. 


Check MSR_PERF_GLOBAL_STATUS MSR (38€H) 
handle any overflow conditions. 


Check OVF flag of each CCCR for overflow 
condition 


Clear overflow status. 


Clear MSR_PERF_GLOBAL_STATUS MSR (38€H) 
using [A32_PERF_GLOBAL_OVF_CTRL MSR (390h). 


Clear OVF flag of each CCCR. 


Write “sample-after” values. 


Configure the counter(s) with the sample after value. 


Configure specific counter 
configuration MSR. 


« Set local enable bit 22 - 1. 
* Do NOT set local counter PMI/INT bit, bit 20 - 0. 
« Event programmed must be PEBS capable. 


= Set appropriate OVF_PMI bits - 1. 
* Only CCCR for MSR_IQ_COUNTER4 
support PEBS. 


Allocate buffer for PEBS states. 


Allocate a buffer in memory for the precise information. 


Program the |[A32_DS_AREA MSR. 


Program the IA32_DS_AREA MSR. 


Configure the PEBS buffer 
management records. 


Configure the PEBS buffer management records in the DS buffer management area. 


Configure/Enable PEBS. 


Set Enable PMCO bit in |A32_PEBS_ENABLE MSR 
(3F1H). 


Configure MSR_PEBS_ENABLE, 
MSR_PEBS_MATRIX_VERT and 
MSR_PEBS_MATRIX_HORZ as needed. 


Enable counters. 


NOTES: 


Set Enable bits in MSR_PERF_GLOBAL_CTRL MSR 
(38FH). 


Set each CCCR enable bit 12 - 1. 


1. Counters read while enabled are not guaranteed to be precise with event counts that occur in timing proximity to the RDMSR. 


18.6.2.4.4 Re-configuring PEBS Facilities 


When software needs to reconfigure PEBS facilities, it should allow a quiescent period between stopping the prior 
event counting and setting up a new PEBS event. The quiescent period is to allow any latent residual PEBS records 
to complete its capture at their previously specified buffer address (provided by IA32_DS_AREA). 
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18.6.3. Performance Monitoring (Processors Based on Intel NetBurst® Microarchitecture) 


The performance monitoring mechanism provided in processors based on Intel NetBurst microarchitecture is 
different from that provided in the P6 family and Pentium processors. While the general concept of selecting, 
filtering, counting, and reading performance events through the WRMSR, RDMSR, and RDPMC instructions is 
unchanged, the setup mechanism and MSR layouts are incompatible with the P6 family and Pentium processor 
mechanisms. Also, the RDPMC instruction has been extended to support faster reading of counters and to read all 
performance counters available in processors based on Intel NetBurst microarchitecture. 


The event monitoring mechanism consists of the following facilities: 


° The IA32_MISC_ENABLE MSR, which indicates the availability in an Intel 64 or IA-32 processor of the 
performance monitoring and processor event-based sampling (PEBS) facilities. 


¢ Event selection control (ESCR) MSRs for selecting events to be monitored with specific performance counters. 
The number available differs by family and model (43 to 45). 


¢ 18 performance counter MSRs for counting events. 


¢ 18 counter configuration control (CCCR) MSRs, with one CCCR associated with each performance counter. 
CCCRs sets up an associated performance counter for a specific method of counting. 


¢ A debug store (DS) save area in memory for storing PEBS records. 
¢ The IA32_DS_AREA MSR, which establishes the location of the DS save area. 


¢ The debug store (DS) feature flag (bit 21) returned by the CPUID instruction, which indicates the availability of 
the DS mechanism. 


¢ The MSR_PEBS_ENABLE MSR, which enables the PEBS facilities and replay tagging used in at-retirement event 
counting. 


° Aset of predefined events and event metrics that simplify the setting up of the performance counters to count 
specific events. 


Table 18-80 lists the performance counters and their associated CCCRs, along with the ESCRs that select events to 
be counted for each performance counter. Predefined event metrics and events can be found at: https://perfmon- 
events.intel.com/. 


Table 18-80. Performance Counter MSRs and Associated CCCR and 
ESCR MSRs (Processors Based on Intel NetBurst Microarchitecture) 


Counter CCCR ESCR 
Name No. Addr Name Addr Name 


MSR_BPU_COUNTERO 0 300H MSR_BPU_CCCRO 360H | MSR_BSU_ESCRO 
MSR_FSB_ESCRO 

MSR_MOB_ESCRO 
MSR_PMH_ESCRO 


2 
° 
> 
a 
a 
— 


WNWRORNON 
Ww 
> 
(=) 
= 


MSR_BPU_ESCRO 3B2H 
MSR_IS_ESCRO 3B4H 
MSR_ITLB_ESCRO 3B6H 
MSR_IX_ESCRO 3C8H 
MSR_BPU_COUNTER1 1 301H MSR_BPU_CCCR1 361H | MSR_BSU_ESCRO 3A0H 


MSR_FSB_ESCRO 

MSR_MOB_ESCRO 
MSR_PMH_ESCRO 
MSR_BPU_ESCRO 


MNWReORRNON 
Ww 
> 
(on) 
a5 


MSR_IS_ESCRO 3B4H 
MSR_ITLB_ESCRO 3B6H 
MSR_IX_ESCRO 3C8H 
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Table 18-80. Performance Counter MSRs and Associated CCCR and 
ESCR MSRs (Processors Based on Intel NetBurst Microarchitecture) (Contd.) 


Counter CCCR ESCR 
Name No. Addr Name Addr Name No. | Addr 
MSR_BPU_COUNTER2 2 302H MSR_BPU_CCCR2 362H MSR_BSU_ESCR1 7 3A1H 
MSR_FSB_ESCR1 6 3A3H 
MSR_MOB_ESCR1 2 3ABH 
MSR_PMH_ESCR1 4 3ADH 
MSR_BPU_ESCR1 0 3B3H 
MSR_IS_ESCR1 1 3B5H 
MSR_ITLB_ESCR1 3 3B7H 
MSR_IX_ESCR1 5 3C9H 
MSR_BPU_COUNTER3 3 303H MSR_BPU_CCCR3 363H MSR_BSU_ESCR1 7 3A1H 
MSR_FSB_ESCR1 6 3A3H 
MSR_MOB_ESCR1 2 3ABH 
MSR_PMH_ESCR1 4 3ADH 
MSR_BPU_ESCR1 0 3B3H 
MSR_IS_ESCR1 1 3B5H 
MSR_ITLB_ESCR1 3 3B7H 
MSR_IX_ESCR1 5 3C9H 
MSR_MS_COUNTERO 4 304H MSR_MS_CCCRO 364H MSR_MS_ESCRO 0 3COH 
MSR_TBPU_ESCRO 2 3C2H 
MSR_TC_ESCRO 1 3C4H 
MSR_MS_COUNTER1 5 305H MSR_MS_CCCR1 365H MSR_MS_ESCRO 0 3COH 
MSR_TBPU_ESCRO 2 3C2H 
MSR_TC_ESCRO 1 3C4H 
MSR_MS_COUNTER2 6 306H MSR_MS_CCCR2 366H MSR_MS_ESCR1 0 3C1H 
MSR_TBPU_ESCR1 2 3C3H 
MSR_TC_ESCR1 1 3C5H 
MSR_MS_COUNTER3 7 307H MSR_MS_CCCR3 367H MSR_MS_ESCR1 0 3C1H 
MSR_TBPU_ESCR1 2 3C3H 
MSR_TC_ESCR1 1 3C5H 
MSR_FLAME_COUNTERO 8 308H MSR_FLAME_CCCRO 368H MSR_FIRM_ESCRO 1 3A4H 
MSR_FLAME_ESCRO 0 3A6H 
MSR_DAC_ESCRO 5 3A8H 
MSR_SAAT_ESCRO 2 3AEH 
MSR_U2L_ESCRO 3 3BOH 
MSR_FLAME_COUNTER1 9 309H MSR_FLAME_CCCR1 369H MSR_FIRM_ESCRO 1 3A4H 
MSR_FLAME_ESCRO 0 3A6H 
MSR_DAC_ESCRO 5 3A8H 
MSR_SAAT_ESCRO 2 3AEH 
MSR_U2L_ESCRO 3 3BOH 
MSR_FLAME_COUNTER2 10 30AH MSR_FLAME_CCCR2 36AH MSR_FIRM_ESCR1 1 3A5H 
MSR_FLAME_ESCR1 0 3A7H 
MSR_DAC_ESCR1 5 3A9H 
MSR_SAAT_ESCR1 2 3AFH 
MSR_U2L_ESCR1 3 3B1H 
MSR_FLAME_COUNTER3 11 30BH MSR_FLAME_CCCR3 36BH MSR_FIRM_ESCR1 1 3A5H 
MSR_FLAME_ESCR1 0 3A7H 
MSR_DAC_ESCR1 5 3A9H 
MSR_SAAT_ESCR1 2 3AFH 
MSR_U2L_ESCR1 3 3B1H 
MSR_IQ_COUNTERO 12 30CH MSR_IQ_CCCRO 36CH MSR_CRU_ESCRO 4 3B8H 
MSR_CRU_ESCR2 5 3CCH 
MSR_CRU_ESCR4 6 3E0H 
MSR_IQ_ESCRO! 0 3BAH 
MSR_RAT_ESCRO 2 3BCH 
MSR_SSU_ESCRO 3 3BEH 
MSR_ALF_ESCRO 1 3CAH 
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ESCR MSRs (Processors Based on Intel NetBurst Microarchitecture) (Contd.) 


Counter CCCR ESCR 

Name No. Addr Name Addr Name No. | Addr 

MSR_IQ_COUNTER1 13 30DH MSR_IQ_CCCR1 36DH MSR_CRU_ESCRO 4 3B8H 
MSR_CRU_ESCR2 5 3CCH 
MSR_CRU_ESCR4 6 3€0H 
MSR_IQ_ESCRO! 0 3BAH 
MSR_RAT_ESCRO 2 3BCH 
MSR_SSU_ESCRO 3 3BEH 
MSR_ALF_ESCRO 1 3CAH 

MSR_IQ_COUNTER2 14 30EH MSR_IQ_CCCR2 36EH MSR_CRU_ESCR1 4 3B9H 
MSR_CRU_ESCR3 5 3CDH 
MSR_CRU_ESCR5 6 3E1H 
MSR_IQ_ESCR1' 0 3BBH 
MSR_RAT_ESCR1 2 3BDH 
MSR_ALF_ESCR1 1 3CBH 

MSR_IQ_COUNTER3 15 30FH MSR_IQ_CCCR3 36FH MSR_CRU_ESCR1 4 3B9H 
MSR_CRU_ESCR3 5 3CDH 
MSR_CRU_ESCR5 6 3E1H 
MSR_IQ_ESCR1! 0 3BBH 
MSR_RAT_ESCR1 2 3BDH 
MSR_ALF_ESCR1 1 3CBH 

MSR_IQ_COUNTER4 16 310H MSR_IQ_CCCR4 370H MSR_CRU_ESCRO 4 3B8H 
MSR_CRU_ESCR2 5 3CCH 
MSR_CRU_ESCR4 6 3€0H 
MSR_IQ_ESCRO! 0 3BAH 
MSR_RAT_ESCRO 2 3BCH 
MSR_SSU_ESCRO 3 3BEH 
MSR_ALF_ESCRO 1 3CAH 

MSR_IQ_COUNTERS 17 311H MSR_IQ_CCCR5 371H MSR_CRU_ESCR1 4 3B9H 
MSR_CRU_ESCR3 5 3CDH 
MSR_CRU_ESCR5 6 3€1H 
MSR_IQ_ESCR1! 0 3BBH 
MSR_RAT_ESCR1 2 3BDH 
MSR_ALF_ESCR1 1 3CBH 


NOTES: 
1. MSR_IQ_ESCRO and MSR_IQ_ESCR1 are available only on early processor builds (family OFH, models 01H-O2H). These MSRs are not 


available on later versions. 


The types of events that can be counted with these performance monitoring facilities are divided into two classes: 
non-retirement events and at-retirement events. 


Non-retirement events are events that occur any time during instruction execution (such as bus transactions or 


cache transactions). 


At-retirement events are events that are counted at the retirement stage of instruction execution, which allows 
finer granularity in counting events and capturing machine state. 


The at-retirement counting mechanism includes facilities for tagging pops that have encountered a particular 
performance event during instruction execution. Tagging allows events to be sorted between those that 
occurred on an execution path that resulted in architectural state being committed at retirement as well as 
events that occurred on an execution path where the results were eventually cancelled and never committed to 


architectural state (such as, the execution of a mispredicted branch). 


The Pentium 4 and Intel Xeon processor performance monitoring facilities support the three usage models 
described below. The first two models can be used to count both non-retirement and at-retirement events; the 


third model is used to count a subset of at-retirement events: 


Event counting — A performance counter is configured to count one or more types of events. While the 
counter is counting, software reads the counter at selected intervals to determine the number of events that 
have been counted between the intervals. 


Vol. 3B 18-103 


PERFORMANCE MONITORING 


Interrupt-based event sampling — A performance counter is configured to count one or more types of 
events and to generate an interrupt when it overflows. To trigger an overflow, the counter is preset to a 
modulus value that will cause the counter to overflow after a specific number of events have been counted. 


When the counter overflows, the processor generates a performance monitoring interrupt (PMI). The interrupt 
service routine for the PMI then records the return instruction pointer (RIP), resets the modulus, and restarts 
the counter. Code performance can be analyzed by examining the distribution of RIPs with a tool like the 
VTune™ Performance Analyzer. 


Processor event-based sampling (PEBS) — In PEBS, the processor writes a record of the architectural 
state of the processor to a memory buffer after the counter overflows. The records of architectural state 
provide additional information for use in performance tuning. Processor-based event sampling can be used to 
count only a subset of at-retirement events. PEBS captures more precise processor state information compared 
to interrupt based event sampling, because the latter need to use the interrupt service routine to re-construct 
the architectural states of processor. 


The following sections describe the MSRs and data structures used for performance monitoring in the Pentium 4 
and Intel Xeon processors. 


18.6.3.1 ESCRMSRs 


The 45 ESCR MSRs (see Table 18-80) allow software to select specific events to be countered. Each ESCR is usually 
associated with a pair of performance counters (see Table 18-80) and each performance counter has several ESCRs 
associated with it (allowing the events counted to be selected from a variety of events). 


Figure 18-45 shows the layout of an ESCR MSR. The functions of the flags and fields are: 


USR flag, bit 2 — When set, events are counted when the processor is operating at a current privilege level 
(CPL) of 1, 2, or 3. These privilege levels are generally used by application code and unprotected operating 
system code. 


OS flag, bit 3 — When set, events are counted when the processor is operating at CPL of 0. This privilege level 
is generally reserved for protected operating system code. (When both the OS and USR flags are set, events 
are counted at all privilege levels.) 


31 30 25 24 98 543210 
Event c Mask Tag 
Select vent Flas Value 
Tag Enable 
Reserved os 
USR 
63 32 
Reserved 


Figure 18-45. Event Selection Control Register (ESCR) for Pentium 4 
and Intel Xeon Processors without Intel HT Technology Support 


Tag enable, bit 4 — When set, enables tagging of wops to assist in at-retirement event counting; when clear, 
disables tagging. See Section 18.6.3.6, “At-Retirement Counting.” 


Tag value field, bits 5 through 8 — Selects a tag value to associate with a pop to assist in at-retirement 
event counting. 


Event mask field, bits 9 through 24 — Selects events to be counted from the event class selected with the 
event select field. 


Event select field, bits 25 through 30) — Selects a class of events to be counted. The events within this 
class that are counted are selected with the event mask field. 
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When setting up an ESCR, the event select field is used to select a specific class of events to count, such as retired 
branches. The event mask field is then used to select one or more of the specific events within the class to be 
counted. For example, when counting retired branches, four different events can be counted: branch not taken 
predicted, branch not taken mispredicted, branch taken predicted, and branch taken mispredicted. The OS and 
USR flags allow counts to be enabled for events that occur when operating system code and/or application code are 
being executed. If neither the OS nor USR flag is set, no events will be counted. 


The ESCRs are initialized to all Os on reset. The flags and fields of an ESCR are configured by writing to the ESCR 
using the WRMSR instruction. Table 18-80 gives the addresses of the ESCR MSRs. 


Writing to an ESCR MSR does not enable counting with its associated performance counter; it only selects the event 
or events to be counted. The CCCR for the selected performance counter must also be configured. Configuration of 
the CCCR includes selecting the ESCR and enabling the counter. 


18.6.3.2 Performance Counters 


The performance counters in conjunction with the counter configuration control registers (CCCRs) are used for 
filtering and counting the events selected by the ESCRs. Processors based on Intel NetBurst microarchitecture 
provide 18 performance counters organized into 9 pairs. A pair of performance counters is associated with a partic- 
ular subset of events and ESCR’s (see Table 18-80). The counter pairs are partitioned into four groups: 


¢ The BPU group, includes two performance counter pairs: 
— MSR_BPU_COUNTERO and MSR_BPU_COUNTER1. 
— MSR_BPU_COUNTER2 and MSR_BPU_COUNTER3. 
¢ The MS group, includes two performance counter pairs: 
— MSR_MS_COUNTERO and MSR_MS_COUNTER1. 
— MSR_MS_COUNTER2 and MSR_MS_COUNTER3. 
¢ The FLAME group, includes two performance counter pairs: 
— MSR_FLAME_COUNTERO and MSR_FLAME_COUNTER1. 
— MSR_FLAME_COUNTER2 and MSR_FLAME_COUNTER3. 
¢ The IQ group, includes three performance counter pairs: 
— MSR_IQ_COUNTERO and MSR_IQ_COUNTER1. 
— MSR_IQ_COUNTER2 and MSR_IQ_COUNTER3. 
— MSR_IQ_COUNTER4 and MSR_IQ_COUNTERS. 
The MSR_IQ_COUNTER4 counter in the IQ group provides support for the PEBS. 


Alternate counters in each group can be cascaded: the first counter in one pair can start the first counter in the 
second pair and vice versa. A similar cascading is possible for the second counters in each pair. For example, within 
the BPU group of counters, MSR_BPU_COUNTERO can start MSR_BPU_COUNTER2 and vice versa, and 
MSR_BPU_COUNTER1 can start MSR_BPU_COUNTER3 and vice versa (see Section 18.6.3.5.6, “Cascading Coun- 
ters”). The cascade flag in the CCCR register for the performance counter enables the cascading of counters. 


Each performance counter is 40-bits wide (see Figure 18-46). The RDPMC instruction is intended to allow reading 
of either the full counter-width (40-bits) or, if ECX[31] is set to 1, the low 32-bits of the counter. Reading the low 
32-bits is faster than reading the full counter width and is appropriate in situations where the count is small enough 
to be contained in 32 bits. In such cases, counter bits 31:0 are written to EAX, while 0 is written to EDX. 


The RDPMC instruction can be used by programs or procedures running at any privilege level and in virtual-8086 
mode to read these counters. The PCE flag in control register CR4 (bit 8) allows the use of this instruction to be 
restricted to only programs and procedures running at privilege level 0. 
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Figure 18-46. Performance Counter (Pentium 4 and Intel Xeon Processors) 


The RDPMC instruction is not serializing or ordered with other instructions. Thus, it does not necessarily wait until 
all previous instructions have been executed before reading the counter. Similarly, subsequent instructions may 
begin execution before the RDPMC instruction operation is performed. 


Only the operating system, executing at privilege level 0, can directly manipulate the performance counters, using 
the RDMSR and WRMSR instructions. A secure operating system would clear the PCE flag during system initializa- 
tion to disable direct user access to the performance-monitoring counters, but provide a user-accessible program- 
ming interface that emulates the RDPMC instruction. 


Some uses of the performance counters require the counters to be preset before counting begins (that is, before 
the counter is enabled). This can be accomplished by writing to the counter using the WRMSR instruction. To set a 
counter to a specified number of counts before overflow, enter a 2s complement negative integer in the counter. 
The counter will then count from the preset value up to -1 and overflow. Writing to a performance counter ina 
Pentium 4 or Intel Xeon processor with the WRMSR instruction causes all 40 bits of the counter to be written. 


18.6.3.3  CCCRMSRs 


Each of the 18 performance counters has one CCCR MSR associated with it (see Table 18-80). The CCCRs control 
the filtering and counting of events as well as interrupt generation. Figure 18-47 shows the layout of an CCCR MSR. 
The functions of the flags and fields are as follows: 


¢ Enable flag, bit 12 — When set, enables counting; when clear, the counter is disabled. This flag is cleared on 
reset. 


° ESCR select field, bits 13 through 15 — Identifies the ESCR to be used to select events to be counted with 
the counter associated with the CCCR. 


¢ Compare flag, bit 18 — When set, enables filtering of the event count; when clear, disables filtering. The 
filtering method is selected with the threshold, complement, and edge flags. 


* Complement flag, bit 19 — Selects how the incoming event count is compared with the threshold value. 
When set, event counts that are less than or equal to the threshold value result in a single count being delivered 
to the performance counter; when clear, counts greater than the threshold value result in a count being 
delivered to the performance counter (see Section 18.6.3.5.2, “Filtering Events”). The complement flag is not 
active unless the compare flag is set. 


¢ Threshold field, bits 20 through 23 — Selects the threshold value to be used for comparisons. The 
processor examines this field only when the compare flag is set, and uses the complement flag setting to 
determine the type of threshold comparison to be made. The useful range of values that can be entered in this 
field depend on the type of event being counted (see Section 18.6.3.5.2, “Filtering Events”). 


° Edge flag, bit 24 — When set, enables rising edge (false-to-true) edge detection of the threshold comparison 
output for filtering event counts; when clear, rising edge detection is disabled. This flag is active only when the 
compare flag is set. 
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Figure 18-47. Counter Configuration Control Register (CCCR) 


FORCE_OVF flag, bit 25 — When set, forces a counter overflow on every counter increment; when clear, 
overflow only occurs when the counter actually overflows. 


OVF_PMI flag, bit 26 — When set, causes a performance monitor interrupt (PMI) to be generated when the 
counter overflows occurs; when clear, disables PMI generation. Note that the PMI is generated on the next 
event count after the counter has overflowed. 


Cascade flag, bit 30 — When set, enables counting on one counter of a counter pair when its alternate 
counter in the other the counter pair in the same counter group overflows (see Section 18.6.3.2, “Performance 
Counters,” for further details); when clear, disables cascading of counters. 


OVF flag, bit 31 — Indicates that the counter has overflowed when set. This flag is a sticky flag that must be 
explicitly cleared by software. 


The CCCRs are initialized to all Os on reset. 


The events that an enabled performance counter actually counts are selected and filtered by the following flags and 
fields in the ESCR and CCCR registers and in the qualification order given: 


1. 


5. 


The event select and event mask fields in the ESCR select a class of events to be counted and one or more 
event types within the class, respectively. 


The OS and USR flags in the ESCR selected the privilege levels at which events will be counted. 


The ESCR select field of the CCCR selects the ESCR. Since each counter has several ESCRs associated with it, 
one ESCR must be chosen to select the classes of events that may be counted. 


The compare and complement flags and the threshold field of the CCCR select an optional threshold to be used 
in qualifying an event count. 


The edge flag in the CCCR allows events to be counted only on rising-edge transitions. 


The qualification order in the above list implies that the filtered output of one “stage” forms the input for the next. 
For instance, events filtered using the privilege level flags can be further qualified by the compare and complement 
flags and the threshold field, and an event that matched the threshold criteria, can be further qualified by edge 
detection. 


The uses of the flags and fields in the CCCRs are discussed in greater detail in Section 18.6.3.5, “Programming the 
Performance Counters for Non-Retirement Events.” 
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18.6.3.4 Debug Store (DS) Mechanism 


The debug store (DS) mechanism was introduced with processors based on Intel NetBurst microarchitecture to 
allow various types of information to be collected in memory-resident buffers for use in debugging and tuning 
programs. The DS mechanism can be used to collect two types of information: branch records and processor event- 
based sampling (PEBS) records. The availability of the DS mechanism in a processor is indicated with the DS 
feature flag (bit 21) returned by the CPUID instruction. 


See Section 17.4.5, “Branch Trace Store (BTS),” and Section 18.6.3.8, “Processor Event-Based Sampling (PEBS),” 
for a description of these facilities. Records collected with the DS mechanism are saved in the DS save area. See 
Section 17.4.9, “BTS and DS Save Area.” 


18.6.3.5 Programming the Performance Counters for Non-Retirement Events 

The basic steps to program a performance counter and to count events include the following: 
1. Select the event or events to be counted. 

2. For each event, select an ESCR that supports the event. 


3. Match the CCCR Select value and ESCR name to a value listed in Table 18-80; select a CCCR and performance 
counter. 


4. Set up an ESCR for the specific event or events to be counted and the privilege levels at which they are to be 
counted. 


5. Set up the CCCR for the performance counter by selecting the ESCR and the desired event filters. 


Set up the CCCR for optional cascading of event counts, so that when the selected counter overflows its 
alternate counter starts. 


7. Set up the CCCR to generate an optional performance monitor interrupt (PMI) when the counter overflows. If 
PMI generation is enabled, the local APIC must be set up to deliver the interrupt to the processor and a handler 
for the interrupt must be in place. 


8. Enable the counter to begin counting. 


18.6.3.5.1 Selecting Events to Count 


There is a set of at-retirement events for processors based on Intel NetBurst microarchitecture. For each event, 
setup information is provided. Table 18-81 gives an example of one of the events. 


Table 18-81. Event Example 
Event Name _ | Event Parameters Parameter Value | Description 


branch_retired Counts the retirement of a branch. Specify one or more mask bits to select 
any combination of branch taken, not-taken, predicted and mispredicted. 


ESCR restrictions MSR_CRU_ESCR2 | See Table 15-3 for the addresses of the ESCR MSRs. 
MSR_CRU_ESCR3 


Counter numbers per | ESCR2: 12, 13, 16 | The counter numbers associated with each ESCR are provided. The 


ESCR ESCR3: 14, 15, 17 | Performance counters and corresponding CCCRs can be obtained from 
Table 15-3. 
ESCR Event Select O6H ESCR[31:25] 
ESCR Event Mask ESCR[24:9] 
Bit 0: MMNP Branch Not-taken Predicted 

1: MMNM Branch Not-taken Mispredicted 

2: MMTP Branch Taken Predicted 

3: MMTM Branch Taken Mispredicted 
CCCR Select 05H CCCR[15:13] 
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Table 18-81. Event Example (Contd.) 


Event Name_| Event Parameters Parameter Value | Description 


Event Specific Notes P6: EMON_BR_INST_RETIRED 
Can Support PEBS No 


Requires Additional | No 
MSRs for Tagging 


Event Parameters are described below. 


ESCR restrictions — Lists the ESCRs that can be used to program the event. Typically only one ESCR is 
needed to count an event. 


Counter numbers per ESCR — Lists which performance counters are associated with each ESCR. Table 18-80 
gives the name of the counter and CCCR for each counter number. Typically only one counter is needed to count 
the event. 


ESCR event select — Gives the value to be placed in the event select field of the ESCR to select the event. 
ESCR event mask — Gives the value to be placed in the Event Mask field of the ESCR to select sub-events to 
be counted. The parameter value column defines the documented bits with relative bit position offset starting 
from 0, where the absolute bit position of relative offset 0 is bit 9 of the ESCR. All undocumented bits are 
reserved and should be set to 0. 

CCCR select — Gives the value to be placed in the ESCR select field of the CCCR associated with the counter 
to select the ESCR to be used to define the event. This value is not the address of the ESCR; it is the number of 
the ESCR from the Number column in Table 18-80. 


Event specific notes — Gives additional information about the event, such as the name of the same or a 
similar event defined for the P6 family processors. 


Can support PEBS — Indicates if PEBS is supported for the event (only supplied for at-retirement events). 


Requires additional MSR for tagging — Indicates which if any additional MSRs must be programmed to 
count the events (only supplied for the at-retirement events). 


NOTE 


The performance-monitoring events found at https://perfmon-events.intel.com/ are intended to 
be used as guides for performance tuning. The counter values reported are not guaranteed to be 
absolutely accurate and should be used as a relative guide for tuning. Known discrepancies are 
documented where applicable. 


The following procedure shows how to set up a performance counter for basic counting; that is, the counter is set 
up to count a specified event indefinitely, wrapping around whenever it reaches its maximum count. This procedure 
is continued through the following four sections. 


An event to be counted can be selected as follows: 


N 


- 


Select the event to be counted. 
Select the ESCR to be used to select events to be counted from the ESCRs field. 
Select the number of the counter to be used to count the event from the Counter Numbers Per ESCR field. 


Determine the name of the counter and the CCCR associated with the counter, and determine the MSR 
addresses of the counter, CCCR, and ESCR from Table 18-80. 


Use the WRMSR instruction to write the ESCR Event Select and ESCR Event Mask values into the appropriate 
fields in the ESCR. At the same time set or clear the USR and OS flags in the ESCR as desired. 


Use the WRMSR instruction to write the CCCR Select value into the appropriate field in the CCCR. 
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NOTE 


Typically all the fields and flags of the CCCR will be written with one WRMSR instruction; however, 
in this procedure, several WRMSR writes are used to more clearly demonstrate the uses of the 
various CCCR fields and flags. 


This setup procedure is continued in the next section, Section 18.6.3.5.2, “Filtering Events.” 


18.6.3.5.2 Filtering Events 


Each counter receives up to 4 input lines from the processor hardware from which it is counting events. The counter 
treats these inputs as binary inputs (input O has a value of 1, input 1 has a value of 2, input 3 has a value of 4, and 
input 3 has a value of 8). When a counter is enabled, it adds this binary input value to the counter value on each 
clock cycle. For each clock cycle, the value added to the counter can then range from 0 (no event) to 15. 


For many events, only the 0 input line is active, so the counter is merely counting the clock cycles during which the 
0 input is asserted. However, for some events two or more input lines are used. Here, the counters threshold 
setting can be used to filter events. The compare, complement, threshold, and edge fields control the filtering of 
counter increments by input value. 


If the compare flag is set, then a “greater than” or a “less than or equal to” comparison of the input value vs. a 
threshold value can be made. The complement flag selects “less than or equal to” (flag set) or “greater than” (flag 
clear). The threshold field selects a threshold value of from 0 to 15. For example, if the complement flag is cleared 
and the threshold field is set to 6, than any input value of 7 or greater on the 4 inputs to the counter will cause the 
counter to be incremented by 1, and any value less than 7 will cause an increment of 0 (or no increment) of the 
counter. Conversely, if the complement flag is set, any value from 0 to 6 will increment the counter and any value 
from 7 to 15 will not increment the counter. Note that when a threshold condition has been satisfied, the input to 
the counter is always 1, not the input value that is presented to the threshold filter. 


The edge flag provides further filtering of the counter inputs when a threshold comparison is being made. The edge 
flag is only active when the compare flag is set. When the edge flag is set, the resulting output from the threshold 
filter (a value of 0 or 1) is used as an input to the edge filter. Each clock cycle, the edge filter examines the last and 
current input values and sends a count to the counter only when it detects a “rising edge” event; that is, a false-to- 
true transition. Figure 18-48 illustrates rising edge filtering. 


The following procedure shows how to configure a CCCR to filter events using the threshold filter and the edge filter. 
This procedure is a continuation of the setup procedure introduced in Section 18.6.3.5.1, “Selecting Events to 
Count.” 


7. (Optional) To set up the counter for threshold filtering, use the WRMSR instruction to write values in the CCCR 
compare and complement flags and the threshold field: 


— Set the compare flag. 
— Set or clear the complement flag for less than or equal to or greater than comparisons, respectively. 
— Enter a value from 0 to 15 in the threshold field. 

8. (Optional) Select rising edge filtering by setting the CCCR edge flag. 

This setup procedure is continued in the next section, Section 18.6.3.5.3, “Starting Event Counting.” 


Processor Clock 


Output from 
Threshold Filter 


Counter Increments 
On Rising Edge 
(False-to-True) 


Figure 18-48. Effects of Edge Filtering 
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18.6.3.5.3 Starting Event Counting 


Event counting by a performance counter can be initiated in either of two ways. The typical way is to set the enable 
flag in the counter’s CCCR. Following the instruction to set the enable flag, event counting begins and continues 
until it is stopped (see Section 18.6.3.5.5, “Halting Event Counting”). 


The following procedural step shows how to start event counting. This step is a continuation of the setup procedure 
introduced in Section 18.6.3.5.2, “Filtering Events.” 


9. To start event counting, use the WRMSR instruction to set the CCCR enable flag for the performance counter. 


This setup procedure is continued in the next section, Section 18.6.3.5.4, “Reading a Performance Counter’s 
Count.” 


The second way that a counter can be started by using the cascade feature. Here, the overflow of one counter auto- 
matically starts its alternate counter (see Section 18.6.3.5.6, “Cascading Counters”). 


18.6.3.5.4 Reading a Performance Counter's Count 


Performance counters can be read using either the RDPMC or RDMSR instructions. The enhanced functions of the 
RDPMC instruction (including fast read) are described in Section 18.6.3.2, “Performance Counters.” These instruc- 
tions can be used to read a performance counter while it is counting or when it is stopped. 


The following procedural step shows how to read the event counter. This step is a continuation of the setup proce- 
dure introduced in Section 18.6.3.5.3, “Starting Event Counting.” 


10. To read a performance counters current event count, execute the RDPMC instruction with the counter number 
obtained from Table 18-80 used as an operand. 


This setup procedure is continued in the next section, Section 18.6.3.5.5, “Halting Event Counting.” 


18.6.3.5.5 Halting Event Counting 


After a performance counter has been started (enabled), it continues counting indefinitely. If the counter overflows 
(goes one count past its maximum count), it wraps around and continues counting. When the counter wraps 
around, it sets its OVF flag to indicate that the counter has overflowed. The OVF flag is a sticky flag that indicates 
that the counter has overflowed at least once since the OVF bit was last cleared. 


To halt counting, the CCCR enable flag for the counter must be cleared. 


The following procedural step shows how to stop event counting. This step is a continuation of the setup procedure 
introduced in Section 18.6.3.5.4, “Reading a Performance Counter’s Count.” 


11. To stop event counting, execute a WRMSR instruction to clear the CCCR enable flag for the performance 
counter. 


To halt a cascaded counter (a counter that was started when its alternate counter overflowed), either clear the 
Cascade flag in the cascaded counter’s CCCR MSR or clear the OVF flag in the alternate counter’s CCCR MSR. 


18.6.3.5.6 Cascading Counters 


As described in Section 18.6.3.2, “Performance Counters,” eighteen performance counters are implemented in 
pairs. Nine pairs of counters and associated CCCRs are further organized as four blocks: BPU, MS, FLAME, and IQ 
(see Table 18-80). The first three blocks contain two pairs each. The IQ block contains three pairs of counters (12 
through 17) with associated CCCRs (MSR_IQ_CCCRO through MSR_IQ_CCCRS5). 


The first 8 counter pairs (0 through 15) can be programmed using ESCRs to detect performance monitoring events. 
Pairs of ESCRs in each of the four blocks allow many different types of events to be counted. The cascade flag in 
the CCCR MSR allows nested monitoring of events to be performed by cascading one counter to a second counter 
located in another pair in the same block (see Figure 18-47 for the location of the flag). 


Counters 0 and 1 form the first pair in the BPU block. Either counter 0 or 1 can be programmed to detect an event 
via MSR_MO B_ESCRO. Counters 0 and 2 can be cascaded in any order, as can counters 1 and 3. It’s possible to set 
up 4 counters in the same block to cascade on two pairs of independent events. The pairing described also applies 
to subsequent blocks. Since the IQ PUB has two extra counters, cascading operates somewhat differently if 16 and 
17 are involved. In the IQ block, counter 16 can only be cascaded from counter 14 (not from 12); counter 14 
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cannot be cascaded from counter 16 using the CCCR cascade bit mechanism. Similar restrictions apply to counter 
17. 


Example 18-1. Counting Events 


Assume a scenario where counter X is set up to count 200 occurrences of event A; then counter Y is set up to count 
400 occurrences of event B. Each counter is set up to count a specific event and overflow to the next counter. In the 
above example, counter X is preset for a count of -200 and counter Y for a count of -400; this setup causes the 
counters to overflow on the 200th and 400th counts respectively. 


Continuing this scenario, counter X is set up to count indefinitely and wraparound on overflow. This is described in 
the basic performance counter setup procedure that begins in Section 18.6.3.5.1, “Selecting Events to Count.” 
Counter Y is set up with the cascade flag in its associated CCCR MSR set to 1 and its enable flag set to 0. 


To begin the nested counting, the enable bit for the counter X is set. Once enabled, counter X counts until it over- 
flows. At this point, counter Y is automatically enabled and begins counting. Thus counter X overflows after 200 
occurrences of event A. Counter Y then starts, counting 400 occurrences of event B before overflowing. When 
performance counters are cascaded, the counter Y would typically be set up to generate an interrupt on overflow. 
This is described in Section 18.6.3.5.8, “Generating an Interrupt on Overflow.” 


The cascading counters mechanism can be used to count a single event. The counting begins on one counter then 
continues on the second counter after the first counter overflows. This technique doubles the number of event 
counts that can be recorded, since the contents of the two counters can be added together. 


18.6.3.5.7 EXTENDED CASCADING 


Extended cascading is a model-specific feature in the Intel NetBurst microarchitecture with CPUID 
DisplayFamily_DisplayModel OF_02, OF_03, OF_04, OF_06. This feature uses bit 11 in CCCRs associated with the IQ 
block. See Table 18-82. 


Table 18-82. CCR Names and Bit Positions 


CCCR Name:Bit Position Bit Name Description 
MSR_IQ_CCCR1|2:11 Reserved 

MSR_IQ_CCCRO:11 CASCNT4INTOO Allow counter 4 to cascade into counter 0 
MSR_IQ_CCCR3:11 CASCNTSINTO3 Allow counter 5 to cascade into counter 3 
MSR_IQ_CCCR4:11 CASCNTSINTO4 Allow counter 5 to cascade into counter 4 
MSR_IQ_CCCR5:11 CASCNT4INTO5 Allow counter 4 to cascade into counter 5 


The extended cascading feature can be adapted to the Interrupt based sampling usage model for performance 
monitoring. However, it is known that performance counters do not generate PMI in cascade mode or extended 
cascade mode due to an erratum. This erratum applies to processors with CPUID DisplayFamily_DisplayModel 
signature of OF_O2. For processors with CPUID DisplayFamily_DisplayModel signature of OF_00 and OF_01, the 
erratum applies to processors with stepping encoding greater than O9H. 


Counters 16 and 17 in the IQ block are frequently used in processor event-based sampling or at-retirement 
counting of events indicating a stalled condition in the pipeline. Neither counter 16 or 17 can initiate the cascading 
of counter pairs using the cascade bit in a CCCR. 


Extended cascading permits performance monitoring tools to use counters 16 and 17 to initiate cascading of two 
counters in the IQ block. Extended cascading from counter 16 and 17 is conceptually similar to cascading other 
counters, but instead of using CASCADE bit of a CCCR, one of the four CASCNTXxINTOy bits is used. 


Example 18-2. Scenario for Extended Cascading 


A usage scenario for extended cascading is to sample instructions retired on logical processor 1 after the first 4096 
instructions retired on logical processor 0. A procedure to program extended cascading in this scenario is outlined 
below: 
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Write the value 0 to counter 12. 


2. Write the value 04000603H to MSR_CRU_ESCRO (corresponding to selecting the NBOGNTAG and NBOGTAG 
event masks with qualification restricted to logical processor 1). 


3. Write the value 04038800H to MSR_IQ_CCCRO. This enables CASCNT4INTOO and OVF_PMI. An ISR can sample 
on instruction addresses in this case (do not set ENABLE, or CASCADE). 


Write the value FFFFFOOOH into counter 16.1. 


5. Write the value 0400060CH to MSR_CRU_ESCR2 (corresponding to selecting the NBOGNTAG and NBOGTAG 
event masks with qualification restricted to logical processor 0). 


6. Write the value 00039000H to MSR_IQ_CCCR4 (set ENABLE bit, but not OVF_PMI). 


Another use for cascading is to locate stalled execution in a multithreaded application. Assume MOB replays in 
thread B cause thread A to stall. Getting a sample of the stalled execution in this scenario could be accomplished 
by: 


1. Set up counter B to count MOB replays on thread B. 


2. Set up counter A to count resource stalls on thread A; set its force overflow bit and the appropriate CASCNTx- 
INTOy bit. 


3. Use the performance monitoring interrupt to capture the program execution data of the stalled thread. 


18.6.3.5.8 Generating an Interrupt on Overflow 


Any performance counter can be configured to generate a performance monitor interrupt (PMI) if the counter over- 
flows. The PMI interrupt service routine can then collect information about the state of the processor or program 
when overflow occurred. This information can then be used with a tool like the Intel® VTune™ Performance 
Analyzer to analyze and tune program performance. 


To enable an interrupt on counter overflow, the OVR_PMI flag in the counter’s associated CCCR MSR must be set. 
When overflow occurs, a PMI is generated through the local APIC. (Here, the performance counter entry in the local 
vector table [LVT] is set up to deliver the interrupt generated by the PMI to the processor.) 


The PMI service routine can use the OVF flag to determine which counter overflowed when multiple counters have 
been configured to generate PMIs. Also, note that these processors mask PMIs upon receiving an interrupt. Clear 
this condition before leaving the interrupt handler. 


When generating interrupts on overflow, the performance counter being used should be preset to value that will 
cause an overflow after a specified number of events are counted plus 1. The simplest way to select the preset 
value is to write a negative number into the counter, as described in Section 18.6.3.5.6, “Cascading Counters.” 
Here, however, if an interrupt is to be generated after 100 event counts, the counter should be preset to minus 100 
plus 1 (-100 + 1), or -99. The counter will then overflow after it counts 99 events and generate an interrupt on the 
next (100th) event counted. The difference of 1 for this count enables the interrupt to be generated immediately 
after the selected event count has been reached, instead of waiting for the overflow to be propagation through the 
counter. 


Because of latency in the microarchitecture between the generation of events and the generation of interrupts on 
overflow, it is sometimes difficult to generate an interrupt close to an event that caused it. In these situations, the 
FORCE_OVF flag in the CCCR can be used to improve reporting. Setting this flag causes the counter to overflow on 
every counter increment, which in turn triggers an interrupt after every counter increment. 


18.6.3.5.9 Counter Usage Guideline 


There are some instances where the user must take care to configure counting logic properly, so that it is not 
powered down. To use any ESCR, even when it is being used just for tagging, (any) one of the counters that the 
particular ESCR (or its paired ESCR) can be connected to should be enabled. If this is not done, 0 counts may 
result. Likewise, to use any counter, there must be some event selected in a corresponding ESCR (other than 
no_event, which generally has a select value of 0). 
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18.6.3.6 At-Retirement Counting 


At-retirement counting provides a means counting only events that represent work committed to architectural 
state and ignoring work that was performed speculatively and later discarded. 


One example of this speculative activity is branch prediction. When a branch misprediction occurs, the results of 
instructions that were decoded and executed down the mispredicted path are canceled. If a performance counter 
was Set up to count all executed instructions, the count would include instructions whose results were canceled as 
well as those whose results committed to architectural state. 


To provide finer granularity in event counting in these situations, the performance monitoring facilities provided in 
the Pentium 4 and Intel Xeon processors provide a mechanism for tagging events and then counting only those 
tagged events that represent committed results. This mechanism is called “at-retirement counting.” 


There are predefined at-retirement events and event metrics that can be used to for tagging events when using at 
retirement counting. The following terminology is used in describing at-retirement counting: 


¢ Bogus, non-bogus, retire — In at-retirement event descriptions, the term “bogus” refers to instructions or 
uops that must be canceled because they are on a path taken from a mispredicted branch. The terms “retired” 
and “non-bogus” refer to instructions or pops along the path that results in committed architectural state 
changes as required by the program being executed. Thus instructions and ops are either bogus or non-bogus, 
but not both. Several of the Pentium 4 and Intel Xeon processors’ performance monitoring events (such as, 
Instruction_Retired and Uops_Retired) can count instructions or pops that are retired based on the character- 
ization of bogus” versus non-bogus. 


* Tagging — Tagging is a means of marking pops that have encountered a particular performance event so they 
can be counted at retirement. During the course of execution, the same event can happen more than once per 
yop and a direct count of the event would not provide an indication of how many pops encountered that event. 


The tagging mechanisms allow a nop to be tagged once during its lifetime and thus counted once at retirement. 
The retired suffix is used for performance metrics that increment a count once per uop, rather than once per 
event. For example, a wop may encounter a cache miss more than once during its life time, but a “Miss Retired” 
metric (that counts the number of retired hops that encountered a cache miss) will increment only once for that 
nop. A “Miss Retired” metric would be useful for characterizing the performance of the cache hierarchy for a 
particular instruction sequence. Details of various performance metrics and how these can be constructed using 
the Pentium 4 and Intel Xeon processors performance events are provided in the Intel Pentium 4 Processor 
Optimization Reference Manual (see Section 1.4, “Related Literature”). 


° Replay — To maximize performance for the common case, the Intel NetBurst microarchitecture aggressively 
schedules pops for execution before all the conditions for correct execution are guaranteed to be satisfied. In 
the event that all of these conditions are not satisfied, wops must be reissued. The mechanism that the Pentium 
4 and Intel Xeon processors use for this reissuing of wops is called replay. Some examples of replay causes are 
cache misses, dependence violations, and unforeseen resource constraints. In normal operation, some number 
of replays is common and unavoidable. An excessive number of replays is an indication of a performance 
problem. 


¢ Assist — When the hardware needs the assistance of microcode to deal with some event, the machine takes 
an assist. One example of this is an underflow condition in the input operands of a floating-point operation. The 
hardware must internally modify the format of the operands in order to perform the computation. Assists clear 
the entire machine of uwops before they begin and are costly. 


18.6.3.6.1 Using At-Retirement Counting 


Processors based on Intel NetBurst microarchitecture allow counting both events and pops that encountered a 
specified event. For a subset of the at-retirement events, a nop may be tagged when it encounters that event. The 
tagging mechanisms can be used in Interrupt-based event sampling, and a subset of these mechanisms can be 
used in PEBS. There are four independent tagging mechanisms, and each mechanism uses a different event to 
count pops tagged with that mechanism: 


¢ Front-end tagging — This mechanism pertains to the tagging of nwops that encountered front-end events (for 
example, trace cache and instruction counts) and are counted with the Front_end_event event. 


¢ Execution tagging — This mechanism pertains to the tagging of wops that encountered execution events (for 
example, instruction types) and are counted with the Execution_Event event. 


18-114 Vol. 3B 


PERFORMANCE MONITORING 


¢ Replay tagging — This mechanism pertains to tagging of hops whose retirement is replayed (for example, a 
cache miss) and are counted with the Replay_event event. Branch mispredictions are also tagged with this 
mechanism. 


° No tags — This mechanism does not use tags. It uses the Instr_retired and the Uops_ retired events. 


Each tagging mechanism is independent from all others; that is, a yop that has been tagged using one mechanism 
will not be detected with another mechanism’s tagged-uop detector. For example, if wops are tagged using the 
front-end tagging mechanisms, the Replay_event will not count those as tagged yops unless they are also tagged 
using the replay tagging mechanism. However, execution tags allow up to four different types of uops to be counted 
at retirement through execution tagging. 


The independence of tagging mechanisms does not hold when using PEBS. When using PEBS, only one tagging 
mechanism should be used at a time. 


Certain kinds of pops that cannot be tagged, including I/O, uncacheable and locked accesses, returns, and far 
transfers. 


There are performance monitoring events that support at-retirement counting: specifically the Front_end_event, 
Execution_event, Replay_event, Inst_retired and Uops_retired events. The following sections describe the tagging 
mechanisms for using these events to tag pop and count tagged uops. 


18.6.3.6.2 Tagging Mechanism for Front_end_event 


The Front_end_event counts pops that have been tagged as encountering any of the following events: 


° yop decode events — Tagging uops for nop decode events requires specifying bits in the ESCR associated with 
the performance-monitoring event, Uop_type. 


¢ Trace cache events — Tagging uops for trace cache events may require specifying certain bits in the 
MSR_TC_PRECISE_EVENT MSR. 


The MSRs that are supported by the front-end tagging mechanism must be set and one or both of the NBOGUS and 
BOGUS bits in the Front_end_event event mask must be set to count events. None of the events currently 
supported requires the use of the MSR_TC_PRECISE_EVENT MSR. 


18.6.3.6.3 Tagging Mechanism For Execution_event 


The execution tagging mechanism differs from other tagging mechanisms in how it causes tagging. One upstream 
ESCR is used to specify an event to detect and to specify a tag value (bits 5 through 8) to identify that event. A 
second downstream ESCR is used to detect nops that have been tagged with that tag value identifier using 
Execution_event for the event selection. 


The upstream ESCR that counts the event must have its tag enable flag (bit 4) set and must have an appropriate 
tag value mask entered in its tag value field. The 4-bit tag value mask specifies which of tag bits should be set for 
a particular pop. The value selected for the tag value should coincide with the event mask selected in the down- 
stream ESCR. For example, if a tag value of 1 is set, then the event mask of NBOGUSO should be enabled, corre- 
spondingly in the downstream ESCR. The downstream ESCR detects and counts tagged pops. The normal (not tag 
value) mask bits in the downstream ESCR specify which tag bits to count. If any one of the tag bits selected by the 
mask is set, the related counter is incremented by one. The tag enable and tag value bits are irrelevant for the 
downstream ESCR used to select the Execution_event. 


The four separate tag bits allow the user to simultaneously but distinctly count up to four execution events at 
retirement. (This applies for interrupt-based event sampling. There are additional restrictions for PEBS as noted in 
Section 18.6.3.8.3, “Setting Up the PEBS Buffer.”) It is also possible to detect or count combinations of events by 
setting multiple tag value bits in the upstream ESCR or multiple mask bits in the downstream ESCR. For example, 
use a tag value of 3H in the upstream ESCR and use NBOGUSO/NBOGUS1 in the downstream ESCR event mask. 


18.6.3.7 Tagging Mechanism for Replay_event 


The replay mechanism enables tagging of wops for a subset of all replays before retirement. Use of the replay 
mechanism requires selecting the type of uop that may experience the replay in the MSR_PEBS_MATRIX_VERT 
MSR and selecting the type of event in the MSR_PEBS_ENABLE MSR. Replay tagging must also be enabled with the 
UOP_Tag flag (bit 24) in the MSR_PEBS_ENABLE MSR. 
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The replay tags defined in Table A-5 also enable Processor Event-Based Sampling (PEBS, see Section 17.4.9). Each 
of these replay tags can also be used in normal sampling by not setting Bit 24 nor Bit 25 in 
IA_32_PEBS_ENABLE_MSR. Each of these metrics requires that the Replay_Event be used to count the tagged 
Lops. 


18.6.3.8 Processor Event-Based Sampling (PEBS) 


The debug store (DS) mechanism in processors based on Intel NetBurst microarchitecture allow two types of infor- 
mation to be collected for use in debugging and tuning programs: PEBS records and BTS records. See Section 
17.4.5, “Branch Trace Store (BTS),” for a description of the BTS mechanism. 


PEBS permits the saving of precise architectural information associated with one or more performance events in 
the precise event records buffer, which is part of the DS save area (see Section 17.4.9, “BTS and DS Save Area”). 
To use this mechanism, a counter is configured to overflow after it has counted a preset number of events. After 
the counter overflows, the processor copies the current state of the general-purpose and EFLAGS registers and 
instruction pointer into a record in the precise event records buffer. The processor then resets the count in the 
performance counter and restarts the counter. When the precise event records buffer is nearly full, an interrupt is 
generated, allowing the precise event records to be saved. A circular buffer is not supported for precise event 
records. 


PEBS is supported only for a subset of the at-retirement events: Execution_event, Front_end_event, and 
Replay_event. Also, PEBS can only be carried out using the one performance counter, the MSR_IQ_COUNTER4 
MSR. 


In processors based on Intel Core microarchitecture, a similar PEBS mechanism is also supported using IA32_PMCO 
and IA32_PERFEVTSELO MSRs (See Section 18.6.2.4). 


18.6.3.8.1 Detection of the Availability of the PEBS Facilities 


The DS feature flag (bit 21) returned by the CPUID instruction indicates (when set) the availability of the DS mech- 
anism in the processor, which supports the PEBS (and BTS) facilities. When this bit is set, the following PEBS facil- 
ities are available: 


¢ The PEBS_UNAVAILABLE flag in the IA32_MISC_ENABLE MSR indicates (when clear) the availability of the 
PEBS facilities, including the MSR_PEBS_ENABLE MSR. 


¢ The enable PEBS flag (bit 24) in the MSR_PEBS_ENABLE MSR allows PEBS to be enabled (set) or disabled 
(clear). 


¢ The IA32_DS_AREA MSR can be programmed to point to the DS save area. 


18.6.3.8.2 Setting Up the DS Save Area 


Section 17.4.9.2, “Setting Up the DS Save Area,” describes how to set up and enable the DS save area. This proce- 
dure is common for PEBS and BTS. 


18.6.3.8.3 Setting Up the PEBS Buffer 


Only the MSR_IQ_COUNTER4 performance counter can be used for PEBS. Use the following procedure to set up the 
processor and this counter for PEBS: 


1. Set up the precise event buffering facilities. Place values in the precise event buffer base, precise event index, 
precise event absolute maximum, and precise event interrupt threshold, and precise event counter reset fields 
of the DS buffer management area (see Figure 17-5) to set up the precise event records buffer in memory. 


Enable PEBS. Set the Enable PEBS flag (bit 24) in MSR_PEBS_ENABLE MSR. 


Set up the MSR_IQ_COUNTER4 performance counter and its associated CCCR and one or more ESCRs for 
PEBS. 
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18.6.3.8.4 Writing a PEBS Interrupt Service Routine 


The PEBS facilities share the same interrupt vector and interrupt service routine (called the DS ISR) with the non- 
precise event-based sampling and BTS facilities. To handle PEBS interrupts, PEBS handler code must be included in 
the DS ISR. See Section 17.4.9.5, “Writing the DS Interrupt Service Routine,” for guidelines for writing the DS ISR. 


18.6.3.8.5 Other DS Mechanism Implications 


The DS mechanism is not available in the SMM. It is disabled on transition to the SMM mode. Similarly the DS 
mechanism is disabled on the generation of a machine check exception and is cleared on processor RESET and 
INIT. 


The DS mechanism is available in real address mode. 


18.6.3.9 Operating System Implications 


The DS mechanism can be used by the operating system as a debugging extension to facilitate failure analysis. 
When using this facility, a 25 to 30 times slowdown can be expected due to the effects of the trace store occurring 
on every taken branch. 


Depending upon intended usage, the instruction pointers that are part of the branch records or the PEBS records 
need to have an association with the corresponding process. One solution requires the ability for the DS specific 
operating system module to be chained to the context switch. A separate buffer can then be maintained for each 
process of interest and the MSR pointing to the configuration area saved and setup appropriately on each context 
switch. 


If the BTS facility has been enabled, then it must be disabled and state stored on transition of the system to a sleep 
state in which processor context is lost. The state must be restored on return from the sleep state. 


It is required that an interrupt gate be used for the DS interrupt as opposed to a trap gate to prevent the generation 
of an endless interrupt loop. 


Pages that contain buffers must have mappings to the same physical address for all processes/logical processors, 
such that any change to CR3 will not change DS addresses. If this requirement cannot be satisfied (that is, the 
feature is enabled on a per thread/process basis), then the operating system must ensure that the feature is 
enabled/disabled appropriately in the context switch code. 


18.6.4 Performance Monitoring and Intel Hyper-Threading Technology in Processors Based 
on Intel NetBurst® Microarchitecture 


The performance monitoring capability of processors based on Intel NetBurst microarchitecture and supporting 
Intel Hyper-Threading Technology is similar to that described in Section 18.6.3. However, the capability is extended 
so that: 


¢ Performance counters can be programmed to select events qualified by logical processor IDs. 
¢ Performance monitoring interrupts can be directed to a specific logical processor within the physical processor. 


The sections below describe performance counters, event qualification by logical processor ID, and special purpose 
bits in ESCRs/CCCRs. They also describe MSR_PEBS_ENABLE, MSR_PEBS_MATRIX_VERT, and 
MSR_TC_PRECISE_EVENT. 


18.6.4.1 ESCRMSRs 
Figure 18-49 shows the layout of an ESCR MSR in processors supporting Intel Hyper-Threading Technology. 


The functions of the flags and fields are as follows: 


° 1T1_USR flag, bit O — When set, events are counted when thread 1 (logical processor 1) is executing at a 
current privilege level (CPL) of 1, 2, or 3. These privilege levels are generally used by application code and 
unprotected operating system code. 
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Figure 18-49. Event Selection Control Register (ESCR) for the Pentium 4 Processor, Intel Xeon Processor 
and Intel Xeon Processor MP Supporting Hyper-Threading Technology 


° 71_OS flag, bit 1 — When set, events are counted when thread 1 (logical processor 1) is executing at CPL of 
0. This privilege level is generally reserved for protected operating system code. (When both the T1_OS and 
T1_USR flags are set, thread 1 events are counted at all privilege levels. ) 


° TO_USR flag, bit 2 — When set, events are counted when thread 0 (logical processor 0) is executing at a CPL 
of 1, 2, or 3. 

° TO_OS flag, bit 3 — When set, events are counted when thread 0 (logical processor 0) is executing at CPL of 
0. (When both the TO_OS and TO_USR flags are set, thread 0 events are counted at all privilege levels.) 

* Tag enable, bit 4 — When set, enables tagging of wops to assist in at-retirement event counting; when clear, 
disables tagging. See Section 18.6.3.6, “At-Retirement Counting.” 

° Tag value field, bits 5 through 8 — Selects a tag value to associate with a pop to assist in at-retirement 
event counting. 

¢ Event mask field, bits 9 through 24 — Selects events to be counted from the event class selected with the 
event select field. 


¢ Event select field, bits 25 through 30) — Selects a class of events to be counted. The events within this 
class that are counted are selected with the event mask field. 


The TO_OS and TO_USR flags and the T1_OS and T1_USR flags allow event counting and sampling to be specified 
for a specific logical processor (0 or 1) within an Intel Xeon processor MP (See also: Section 8.4.5, “Identifying 
Logical Processors in an MP System,” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 3A). 


Not all performance monitoring events can be detected within an Intel Xeon processor MP on a per logical processor 
basis (see Section 18.6.4.4, “Performance Monitoring Events”). Some sub-events (specified by an event mask bits) 
are counted or sampled without regard to which logical processor is associated with the detected event. 


18.6.4.2 CCCRMSRs 


Figure 18-50 shows the layout of a CCCR MSR in processors supporting Intel Hyper-Threading Technology. The 
functions of the flags and fields are as follows: 


¢ Enable flag, bit 12 — When set, enables counting; when clear, the counter is disabled. This flag is cleared on 
reset 


° ESCR select field, bits 13 through 15 — Identifies the ESCR to be used to select events to be counted with 
the counter associated with the CCCR. 


° Active thread field, bits 16 and 17 — Enables counting depending on which logical processors are active 
(executing a thread). This field enables filtering of events based on the state (active or inactive) of the logical 
processors. The encodings of this field are as follows: 


00 — None. Count only when neither logical processor is active. 
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01 — Single. Count only when one logical processor is active (either 0 or 1). 

10 — Both. Count only when both logical processors are active. 

11 — Any. Count when either logical processor is active. 

A halted logical processor or a logical processor in the “wait for SIPI” state is considered inactive. 


Compare flag, bit 18 — When set, enables filtering of the event count; when clear, disables filtering. The 
filtering method is selected with the threshold, complement, and edge flags. 


Reserved 
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Figure 18-50. Counter Configuration Control Register (CCCR) 


Complement flag, bit 19 — Selects how the incoming event count is compared with the threshold value. 
When set, event counts that are less than or equal to the threshold value result in a single count being 
delivered to the performance counter; when clear, counts greater than the threshold value result in a count 
being delivered to the performance counter (see Section 18.6.3.5.2, “Filtering Events”). The compare flag is 
not active unless the compare flag is set. 


Threshold field, bits 20 through 23 — Selects the threshold value to be used for comparisons. The 
processor examines this field only when the compare flag is set, and uses the complement flag setting to 
determine the type of threshold comparison to be made. The useful range of values that can be entered in this 
field depend on the type of event being counted (see Section 18.6.3.5.2, “Filtering Events”). 


Edge flag, bit 24 — When set, enables rising edge (false-to-true) edge detection of the threshold comparison 
output for filtering event counts; when clear, rising edge detection is disabled. This flag is active only when the 
compare flag is set. 


FORCE_OVF flag, bit 25 — When set, forces a counter overflow on every counter increment; when clear, 
overflow only occurs when the counter actually overflows. 


OVF_PMI_TO flag, bit 26 — When set, causes a performance monitor interrupt (PMI) to be sent to logical 
processor 0 when the counter overflows occurs; when clear, disables PMI generation for logical processor 0. 
Note that the PMI is generate on the next event count after the counter has overflowed. 


OVF_PMI_T1 flag, bit 27 — When set, causes a performance monitor interrupt (PMI) to be sent to logical 
processor 1 when the counter overflows occurs; when clear, disables PMI generation for logical processor 1. 
Note that the PMI is generate on the next event count after the counter has overflowed. 


Cascade flag, bit 30 — When set, enables counting on one counter of a counter pair when its alternate 
counter in the other the counter pair in the same counter group overflows (see Section 18.6.3.2, “Performance 
Counters,” for further details); when clear, disables cascading of counters. 
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° OVF flag, bit 31 — Indicates that the counter has overflowed when set. This flag is a sticky flag that must be 
explicitly cleared by software. 


18.6.4.3 IA32_PEBS_ENABLE MSR 


In a processor supporting Intel Hyper-Threading Technology and based on the Intel NetBurst microarchitecture, 
PEBS is enabled and qualified with two bits in the MSR_PEBS_ENABLE MSR: bit 25 (ENABLE_PEBS_MY_THR) and 
26 (ENABLE_PEBS_OTH_THR) respectively. These bits do not explicitly identify a specific logical processor by logic 
processor ID(TO or T1); instead, they allow a software agent to enable PEBS for subsequent threads of execution 
on the same logical processor on which the agent is running (“my thread”) or for the other logical processor in the 
physical package on which the agent is not running (“other thread”). 


PEBS is supported for only a subset of the at-retirement events: Execution_event, Front_end_event, and 
Replay_event. Also, PEBS can be carried out only with two performance counters: MSR_IQ_CCCR4 (MSR address 
370H) for logical processor 0 and MSR_IQ_CCCRS5 (MSR address 371H) for logical processor 1. 


Performance monitoring tools should use a processor affinity mask to bind the kernel mode components that need 
to modify the ENABLE_PEBS_MY_THR and ENABLE_PEBS_OTH_THR bits in the MSR_PEBS_ENABLE MSR to a 
specific logical processor. This is to prevent these kernel mode components from migrating between different 
logical processors due to OS scheduling. 


18.6.4.4 Performance Monitoring Events 


When Intel Hyper-Threading Technology is active, many performance monitoring events can be can be qualified by 

the logical processor ID, which corresponds to bit 0 of the initial APIC ID. This allows for counting an event in any 

or all of the logical processors. However, not all the events have this logic processor specificity, or thread specificity. 

Here, each event falls into one of two categories: 

° Thread specific (TS) — The event can be qualified as occurring on a specific logical processor. 

¢ Thread independent (TI) — The event cannot be qualified as being associated with a specific logical 
processor. 


If for example, a TS event occurred in logical processor TO, the counting of the event (as shown in Table 18-83) 
depends only on the setting of the TO_USR and TO_OS flags in the ESCR being used to set up the event counter. 
The T1i_USR and T1_OS flags have no effect on the count. 


Table 18-83. Effect of Logical Processor and CPL Qualification 
for Logical-Processor-Specific (TS) Events 


TO_OS/TO_USR = 00 


T1_OS/T1_USR = 00 


Zero count 


T1_OS/T1_USR = 01 
Counts while T1 in USR 


T1_OS/T1_USR = 11 


Counts while T1 in OS or 
USR 


T1_OS/T1_USR = 10 
Counts while T1 in OS 


TO_OS/TO_USR = 01 


Counts while TO in USR 


Counts while TO in USR 
or T1 in USR 


Counts while (a) TO in 
USR or (b) T1 in OS or (c) 
T1 in USR 


Counts while (a) TO in OS 
or (b) T1 in OS 


TO_OS/TO_USR = 11 


Counts while TO in OS or 
USR 


Counts while (a) TO in OS 
or (b) TO in USR or (c) T1 
in USR 


Counts irrespective of 
CPL, TO, T1 


Counts while (a) TO in OS 
or (b) or TO in USR or (c) 
T1 in OS 


TO_OS/TO_USR = 10 


Counts TO in OS 


Counts TO in OS or T1 in 
USR 


Counts while (a)TO in Os 
or (b) T1 in OS or (c) T1 
in USR 


Counts while (a) TO in OS 
or (b) T1 in OS 


When a bit in the event mask field is TI, the effect of specifying bit-0-3 of the associated ESCR are described in 
Table 15-6. For events that are marked as TI, the effect of selectively specifying TO_USR, TO_OS, Ti1_USR, T1_OS 
bits is shown in Table 18-84. 
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Table 18-84. Effect of Logical Processor and CPL Qualification 
for Non-logical-Processor-specific (Tl) Events 


T1_OS/T1_USR = 00 T1_OS/T1_USR = 01 T1_OS/T1_USR = 11 T1_OS/T1_USR = 10 
TO_OS/TO_USR = 00 Zero count Counts while (a) TO in Counts irrespective of Counts while (a) TO in OS 
USR or (b) T1 in USR CPL, TO, T1 or (b) T1 in OS 
TO_OS/TO_USR = 01 Counts while (a) TO in Counts while (a) TO in Counts irrespective of Counts irrespective of 
USR or (b) T1 in USR USR or (b) T1 in USR CPL, TO, T1 CPL, TO, T1 
TO_OS/TO_USR = 11 Counts irrespective of Counts irrespective of Counts irrespective of Counts irrespective of 
CPL, TO, T1 CPL, TO, T1 CPL, TO, T1 CPL, TO, T1 
TO_OS/TO_USR = 0 Counts while (a) TO in OS | Counts irrespective of Counts irrespective of Counts while (a) TO in OS 
or (b) T1 in OS CPL, TO, T1 CPL, TO, T1 or (b) T1 in OS 


18.6.4.5 Counting Clocks on systems with Intel Hyper-Threading Technology in Processors Based on 
Intel NetBurst® Microarchitecture 


18.6.4.5.1  Non-Halted Clockticks 


Use the following procedure to program ESCRs and CCCRs to obtain non-halted clockticks on processors based on 
Intel NetBurst microarchitecture: 


1. Select an ESCR for the global_power_events and specify the RUNNING sub-event mask and the desired 
TO_OS/TO_USR/T1_OS/T1_USR bits for the targeted processor. 


Select an appropriate counter. 
Enable counting in the CCCR for that counter by setting the enable bit. 


18.6.4.5.2 Non-Sleep Clockticks 


Performance monitoring counters can be configured to count clockticks whenever the performance monitoring 
hardware is not powered-down. To count Non-sleep Clockticks with a performance-monitoring counter, do the 
following: 


1. Select one of the 18 counters. 


2. Select any of the ESCRs whose events the selected counter can count. Set its event select to anything other 
than “no_event”; the counter may be disabled if this is not done. 


3. Turn threshold comparison on in the CCCR by setting the compare bit to “1”. 


Set the threshold to “15” and the complement to “1” in the CCCR. Since no event can exceed this threshold, the 
threshold condition is met every cycle and the counter counts every cycle. Note that this overrides any qualifi- 
cation (e.g. by CPL) specified in the ESCR. 


5. Enable counting in the CCCR for the counter by setting the enable bit. 


In most cases, the counts produced by the non-halted and non-sleep metrics are equivalent if the physical package 
supports one logical processor and is not placed in a power-saving state. Operating systems may execute an HLT 
instruction and place a physical processor in a power-saving state. 


On processors that support Intel Hyper-Threading Technology (Intel HT Technology), each physical package can 
support two or more logical processors. Current implementation of Intel HT Technology provides two logical 
processors for each physical processor. While both logical processors can execute two threads simultaneously, one 
logical processor may halt to allow the other logical processor to execute without sharing execution resources 
between two logical processors. 


Non-halted Clockticks can be set up to count the number of processor clock cycles for each logical processor when- 
ever the logical processor is not halted (the count may include some portion of the clock cycles for that logical 
processor to complete a transition to a halted state). Physical processors that support Intel HT Technology enter 
into a power-saving state if all logical processors halt. 
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The Non-sleep Clockticks mechanism uses a filtering mechanism in CCCRs. The mechanism will continue to incre- 
ment as long as one logical processor is not halted or in a power-saving state. Applications may cause a processor 
to enter into a power-saving state by using an OS service that transfers control to an OS's idle loop. The idle loop 
then may place the processor into a power-saving state after an implementation-dependent period if there is no 
work for the processor. 


18.6.5 Performance Monitoring and Dual-Core Technology 


The performance monitoring capability of dual-core processors duplicates the microarchitectural resources of a 
single-core processor implementation. Each processor core has dedicated performance monitoring resources. 


In the case of Pentium D processor, each logical processor is associated with dedicated resources for performance 
monitoring. In the case of Pentium processor Extreme edition, each processor core has dedicated resources, but 
two logical processors in the same core share performance monitoring resources (see Section 18.6.4, “Perfor- 
mance Monitoring and Intel Hyper-Threading Technology in Processors Based on Intel NetBurst® Microarchitec- 
ture”). 


18.6.6 Performance Monitoring on 64-bit Intel Xeon Processor MP with Up to 8-MByte L3 
Cache 


The 64-bit Intel Xeon processor MP with up to 8-MByte L3 cache has a CPUID signature of family [OFH], model [03H 
or 04H]. Performance monitoring capabilities available to Pentium 4 and Intel Xeon processors with the same 
values (see Section 18.1 and Section 18.6.4) apply to the 64-bit Intel Xeon processor MP with an L3 cache. 


The level 3 cache is connected between the system bus and IOQ through additional control logic. See Figure 18-51. 


System Bus 
¢ i > 


iBUSQ and iSNPQ <—— 


3rd Level Cache 
|__| 
8 or 4 -way 


iFSB 


Processor Core 
10Q |<» (Front end, Execution, 
Retirement, L1, L2 


Figure 18-51. Block Diagram of 64-bit Intel Xeon Processor MP with 8-MByte L3 


Additional performance monitoring capabilities and facilities unique to 64-bit Intel Xeon processor MP with an L3 
cache are described in this section. The facility for monitoring events consists of a set of dedicated model-specific 
registers (MSRs), each dedicated to a specific event. Programming of these MSRs requires using RDMSR/WRMSR 
instructions with 64-bit values. 


The lower 32-bits of the MSRs at addresses 107CC through 107D3 are treated as 32 bit performance counter regis- 
ters. These performance counters can be accessed using RDPMC instruction with the index starting from 18 
through 25. The EDX register returns zero when reading these 8 PMCs. 


The performance monitoring capabilities consist of four events. These are: 
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IBUSQ event — This event detects the occurrence of micro-architectural conditions related to the iBUSQ unit. 
It provides two MSRs: MSR_IFSB_IBUSQO and MSR_IFSB_IBUSQ1. Configure sub-event qualification and 
enable/disable functions using the high 32 bits of these MSRs. The low 32 bits act as a 32-bit event counter. 
Counting starts after software writes a non-zero value to one or more of the upper 32 bits. See Figure 18-52. 


MSR_IFSB_IBUSQx, Addresses: 107CCH and 107CDH 
63 60 59 58 57 56 55 4948 46.45 38 37 36 35 34 33 3231 ) 


1/1 
32 bit event count 


Saturate 
Fill_match ————~ Reserved 
Eviction_match ———— 
L3_state_match 
Snoop_match 
Type_match 
T1_match 
TO_match 


Figure 18-52. MSR_IFSB_IBUSQx, Addresses: 107CCH and 107CDH 


ISNPQ event — This event detects the occurrence of microarchitectural conditions related to the iSNPQ unit. 
It provides two MSRs: MSR_IFSB_ISNPQO and MSR_IFSB_ISNPQ1. Configure sub-event qualifications and 
enable/disable functions using the high 32 bits of the MSRs. The low 32-bits act as a 32-bit event counter. 
Counting starts after software writes a non-zero value to one or more of the upper 32-bits. See Figure 18-53. 


MSR_IFSB_ISNPQx, Addresses: 107CEH and 107CFH Reserved 
63 60 59 58 57 56 55 48 46 45 39 38 37 36 35 34 33 32 


Saturate 
L3_state_match 


Snoop_match 
Type_match 
Agent_match 
T1_match 
TO_match 

31 0 


32 bit event count 


Figure 18-53. MSR_IFSB_ISNPQx, Addresses: 107CEH and 107CFH 


EFSB event — This event can detect the occurrence of micro-architectural conditions related to the iFSB unit 
or system bus. It provides two MSRs: MSR_EFSB_DRDYO and MSR_EFSB_DRDY1. Configure sub-event qualifi- 
cations and enable/disable functions using the high 32 bits of the 64-bit MSR. The low 32-bit act as a 32-bit 
event counter. Counting starts after software writes a non-zero value to one or more of the qualification bits in 
the upper 32-bits of the MSR. See Figure 18-54. 
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MSR_EFSB_DRDYx, Addresses: 107D0H and 107D1H 


63 60 59 58 57 56 55 50 49 48 39 38 37 36 35 34 33 32 
Saturate 
Other Reserved 
Own 
31 0 


32 bit event count 


Figure 18-54. MSR_EFSB_DRDYx, Addresses: 107D0H and 107D1H 


° IBUSQ Latency event — This event accumulates weighted cycle counts for latency measurement of transac- 
tions in the iBUSQ unit. The count is enabled by setting MSR_IFSB_CTRL6[bit 26] to 1; the count freezes after 
software sets MSR_IFSB_CTRL6[bit 26] to 0. MSR_IFSB_CNTR7 acts as a 64-bit event counter for this event. 
See Figure 18-55. 


MSR_IFSB_CTL6 Address: 107D2H 


63 59 57 0 
Enable 
MSR_IFSB_CNTR7 Address: 107D3H Reserved 
63 ) 


64 bit event count 


Figure 18-55. MSR_IFSB_CTL6, Address: 107D2H; 
MSR_IFSB_CNTR7, Address: 107D3H 


18.6.7. Performance Monitoring on L3 and Caching Bus Controller Sub-Systems 


The Intel Xeon processor 7400 series and Dual-Core Intel Xeon processor 7100 series employ a distinct L3/caching 
bus controller sub-system. These sub-system have a unique set of performance monitoring capability and 
programming interfaces that are largely common between these two processor families. 


Intel Xeon processor 7400 series are based on 45 nm enhanced Intel Core microarchitecture. The CPUID signature 
is indicated by DisplayFamily_DisplayModel value of 06_1DH (see CPUID instruction in Chapter 3, “Instruction Set 
Reference, A-L” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A). Intel Xeon 
processor 7400 series have six processor cores that share an L3 cache. 


Dual-Core Intel Xeon processor 7100 series are based on Intel NetBurst microarchitecture, have a CPUID signature 
of family [OFH], model [06H] and a unified L3 cache shared between two cores. Each core in an Intel Xeon 
processor 7100 series supports Intel Hyper-Threading Technology, providing two logical processors per core. 


Both Intel Xeon processor 7400 series and Intel Xeon processor 7100 series support multi-processor configurations 
using system bus interfaces. In Intel Xeon processor 7400 series, the L3/caching bus controller sub-system 
provides three Simple Direct Interface (SDI) to service transactions originated the XQ-replacement SDI logic in 
each dual-core modules. In Intel Xeon processor 7100 series, the IOQ logic in each processor core is replaced with 
a Simple Direct Interface (SDI) logic. The L3 cache is connected between the system bus and the SDI through addi- 
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tional control logic. See Figure 18-56 for the block configuration of six processor cores and the L3/Caching bus 
controller sub-system in Intel Xeon processor 7400 series. Figure 18-56 shows the block configuration of two 
processor cores (four logical processors) and the L3/Caching bus controller sub-system in Intel Xeon processor 


7100 series. 


GBSQ, GSNPQ, 
GINTQ, ... 


SDI 


SDI interface 


L2 


Core Core 


SDI interface 


L2 


Core 


Core 


SDI interface 


L2 


Core 


Core 


Figure 18-56. Block Diagram of Intel Xeon Processor 7400 Series 


Almost all of the performance monitoring capabilities available to processor cores with the same CPUID signatures 
(see Section 18.1 and Section 18.6.4) apply to Intel Xeon processor 7100 series. The MSRs used by performance 


monitoring interface are shared between two logical processors in the same processor core. 


The performance monitoring capabilities available to processor with DisplayFamily_DisplayModel signature 06_17H 
also apply to Intel Xeon processor 7400 series. Each processor core provides its own set of MSRs for performance 


monitoring interface. 


The I0Q_allocation and IOQ_active_entries events are not supported in Intel Xeon processor 7100 series and 7400 
series. Additional performance monitoring capabilities applicable to the L3/caching bus controller sub-system are 


described in this section. 
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Figure 18-57. Block Diagram of Intel Xeon Processor 7100 Series 


18.6.7.1 Overview of Performance Monitoring with L3/Caching Bus Controller 


The facility for monitoring events consists of a set of dedicated model-specific registers (MSRs). There are eight 
event select/counting MSRs that are dedicated to counting events associated with specified microarchitectural 
conditions. Programming of these MSRs requires using RDMSR/WRMSR instructions with 64-bit values. In addition, 
an MSR MSR_EMON_L3_GL_CTL provides simplified interface to control freezing, resetting, re-enabling operation 
of any combination of these event select/counting MSRs. 


The eight MSRs dedicated to count occurrences of specific conditions are further divided to count three sub-classes 

of microarchitectural conditions: 

* Two MSRs (MSR_EMON_L3_CTR_CTLO and MSR_EMON_L3_CTR_CTL1) are dedicated to counting GBSQ 
events. Up to two GBSQ events can be programmed and counted simultaneously. 

¢ Two MSRs (MSR_EMON_L3_CTR_CTL2 and MSR_EMON_L3_CTR_CTL3) are dedicated to counting GSNPQ 
events. Up to two GBSQ events can be programmed and counted simultaneously. 


° Four MSRs (MSR_EMON_L3_CTR_CTL4, MSR_EMON_L3_CTR_CTL5, MSR_EMON_L3_CTR_CTL6, and 
MSR_EMON_L3_CTR_CTL7) are dedicated to counting external bus operations. 


The bit fields in each of eight MSRs share the following common characteristics: 


° Bits 63:32 is the event control field that includes an event mask and other bit fields that control counter 
operation. The event mask field specifies details of the microarchitectural condition, and its definition differs 
across GBSQ, GSNPQ, FSB. 


¢ Bits 31:0 is the event count field. If the specified condition is met during each relevant clock domain of the 
event logic, the matched condition signals the counter logic to increment the associated event count field. The 
lower 32-bits of these 8 MSRs at addresses 107CC through 107D3 are treated as 32 bit performance counter 
registers. 


In Dual-Core Intel Xeon processor 7100 series, the uncore performance counters can be accessed using RDPMC 
instruction with the index starting from 18 through 25. The EDX register returns zero when reading these 8 PMCs. 


In Intel Xeon processor 7400 series, RDPMC with ECX between 2 and 9 can be used to access the eight uncore 
performance counter/control registers. 
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18.6.7.2 GBSQ Event Interface 


The layout of MSR_EMON_L3_CTR_CTLO and MSR_EMON_L3_CTR_CTL1 is given in Figure 18-58. Counting starts 

after software writes a non-zero value to one or more of the upper 32 bits. 

The event mask field (bits 58:32) consists of the following eight attributes: 

e Agent_Select (bits 35:32): The definition of this field differs slightly between Intel Xeon processor 7100 and 
7400. 


For Intel Xeon processor 7100 series, each bit specifies a logical processor in the physical package. The lower 
two bits corresponds to two logical processors in the first processor core, the upper two bits corresponds to two 
logical processors in the second processor core. OFH encoding matches transactions from any logical processor. 


For Intel Xeon processor 7400 series, each bit of [34:32] specifies the SDI logic of a dual-core module as the 
originator of the transaction. A value of 0111B in bits [35:32] specifies transaction from any processor core. 


MSR_EMON_L3_CTR_CTLO/1, Addresses: 107CCH/107CDH Reserved 
63 60 59 58 57 56 55 54 53 4746 44 43 38 37 36 35 32 


Saturate 
Cross_snoop 
Fill_eviction 
Core_module_select ——~ 
L3_state 
Snoop_match 
Type_match 
Data_flow 
Agent_select 


31 0 


32 bit event count 


Figure 18-58. MSR_EMON_L3_CTR_CTLO/1, Addresses: 107CCH/107CDH 


* Data_Flow (bits 37:36): Bit 36 specifies demand transactions, bit 37 specifies prefetch transactions. 


° Type_Match (bits 43:38): Specifies transaction types. If all six bits are set, event count will include all 
transaction types. 


¢ Snoop_Match: (bits 46:44): The three bits specify (in ascending bit position) clean snoop result, HIT snoop 
result, and HITM snoop results respectively. 


° L3_State (bits 53:47): Each bit specifies an L2 coherency state. 


* Core_Module_Select (bits 55:54): The valid encodings for L3 lookup differ slightly between Intel Xeon 
processor 7100 and 7400. 


For Intel Xeon processor 7100 series, 

— OOB: Match transactions from any core in the physical package 

— 01B: Match transactions from this core only 

— 10B: Match transactions from the other core in the physical package 

— 11B: Match transaction from both cores in the physical package 

For Intel Xeon processor 7400 series, 

— OOB: Match transactions from any dual-core module in the physical package 

— 01B: Match transactions from this dual-core module only 

— 10B: Match transactions from either one of the other two dual-core modules in the physical package 
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— 11B: Match transaction from more than one dual-core modules in the physical package 
¢  Fill_Eviction (bits 57:56): The valid encodings are 
— OOB: Match any transactions 
— 01B: Match transactions that fill L3 
— 10B: Match transactions that fill L3 without an eviction 
— 11B: Match transaction fill L3 with an eviction 
* Cross_Snoop (bit 58): The encodings are 
— OB: Match any transactions 
— 1B: Match cross snoop transactions 
For each counting clock domain, if all eight attributes match, event logic signals to increment the event count field. 


18.6.7.3. GSNPQ Event Interface 


The layout of MSR_EMON_L3_CTR_CTL2 and MSR_EMON_L3_CTR_CTL3 is given in Figure 18-59. Counting starts 
after software writes a non-zero value to one or more of the upper 32 bits. 


The event mask field (bits 58:32) consists of the following six attributes: 


¢ Agent_Select (bits 37:32): The definition of this field differs slightly between Intel Xeon processor 7100 and 
7400. 


¢ For Intel Xeon processor 7100 series, each of the lowest 4 bits specifies a logical processor in the physical 
package. The lowest two bits corresponds to two logical processors in the first processor core, the next two bits 
corresponds to two logical processors in the second processor core. Bit 36 specifies other symmetric agent 
transactions. Bit 37 specifies central agent transactions. 3FH encoding matches transactions from any logical 
processor. 


For Intel Xeon processor 7400 series, each of the lowest 3 bits specifies a dual-core module in the physical 
package. Bit 37 specifies central agent transactions. 


° Type_Match (bits 43:38): Specifies transaction types. If all six bits are set, event count will include any 
transaction types. 


¢ Snoop_Match: (bits 46:44): The three bits specify (in ascending bit position) clean snoop result, HIT snoop 
result, and HITM snoop results respectively. 


¢ L2_State (bits 53:47): Each bit specifies an L3 coherency state. 


* Core_Module_Select (bits 56:54): Bit 56 enables Core_Module_Select matching. If bit 56 is clear, 
Core_Module_Select encoding is ignored. The valid encodings for the lower two bits (bit 55, 54) differ slightly 
between Intel Xeon processor 7100 and 7400. 


For Intel Xeon processor 7100 series, if bit 56 is set, the valid encodings for the lower two bits (bit 55, 54) are 
— OOB: Match transactions from only one core (irrespective which core) in the physical package 

— 01B: Match transactions from this core and not the other core 

— 10B: Match transactions from the other core in the physical package, but not this core 

— 11B: Match transaction from both cores in the physical package 

For Intel Xeon processor 7400 series, if bit 56 is set, the valid encodings for the lower two bits (bit 55, 54) are 


— OOB: Match transactions from only one dual-core module (irrespective which module) in the physical 
package. 


— 01B: Match transactions from one or more dual-core modules. 
— 10B: Match transactions from two or more dual-core modules. 
— 11B: Match transaction from all three dual-core modules in the physical package. 
* Block_Snoop (bit 57): specifies blocked snoop. 
For each counting clock domain, if all six attributes match, event logic signals to increment the event count field. 
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MSR_EMON_L3_CTR_CTL2/3, Addresses: 107CEH/107CFH Reserved 
63 60 59 58 57 56 5554 53 4746 44 43 39 38 37 36 32 


Saturate 
Block_snoop -———! 
Core_select 

L2_state 
Snoop_match 
Type_match 
Agent_match 


31 0 


32 bit event count 


Figure 18-59. MSR_EMON_L3_CTR_CTL2/3, Addresses: 107CEH/107CFH 


18.6.7.4 FSB Event Interface 


The layout of MSR_EMON_L3_CTR_CTL4 through MSR_EMON_L3_CTR_CTL7 is given in Figure 18-60. Counting 
starts after software writes a non-zero value to one or more of the upper 32 bits. 

The event mask field (bits 58:32) is organized as follows: 

¢ Bit 58: must set to 1. 

¢ FSB_Submask (bits 57:32): Specifies FSB-specific sub-event mask. 


The FSB sub-event mask defines a set of independent attributes. The event logic signals to increment the associ- 
ated event count field if one of the attribute matches. Some of the sub-event mask bit counts durations. A duration 
event increments at most once per cycle. 


MSR_EMON_L3_CTR_CTL4/5/6/7, Addresses: 107D0H-107D3H 
63 60 59 58 57 56 55 50 49 48 39 38 37 36 35 34 33 32 


1 


Saturate Reserved 


FSB submask 


31 0 


32 bit event count 


Figure 18-60. MSR_EMON_L3_CTR_CTL4/5/6/7, Addresses: 107D0H-107D3H 


18.6.7.4.1 FSB Sub-Event Mask Interface 
° FSB_type (bit 37:32): Specifies different FSB transaction types originated from this physical package. 


e FSB_L_clear (bit 38): Count clean snoop results from any source for transaction originated from this physical 
package. 


¢ ~=FSB_L_hit (bit 39): Count HIT snoop results from any source for transaction originated from this physical 
package. 
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¢ FSB_L_hitm (bit 40): Count HITM snoop results from any source for transaction originated from this physical 
package. 


¢ FSB_L_defer (bit 41): Count DEFER responses to this processor’s transactions. 

¢ FSB_L_retry (bit 42): Count RETRY responses to this processor's transactions. 

¢ FSB_L_snoop_stall (bit 43): Count snoop stalls to this processor's transactions. 

¢ FSB_DBSY (bit 44): Count DBSY assertions by this processor (without a concurrent DRDY). 

¢ FSB_DRDY (bit 45): Count DRDY assertions by this processor. 

¢ FSB_BNR (bit 46): Count BNR assertions by this processor. 

° FSB_IOQ_empty (bit 47): Counts each bus clocks when the IOQ is empty. 

° FSB_IOQ_ full (bit 48): Counts each bus clocks when the I0Q is full. 

° FSB_IOQ_active (bit 49): Counts each bus clocks when there is at least one entry in the IOQ. 

¢ FSB_WW_data (bit 50): Counts back-to-back write transaction’s data phase. 

° FSB_WW_ issue (bit 51): Counts back-to-back write transaction request pairs issued by this processor. 

° FSB_WR_issue (bit 52): Counts back-to-back write-read transaction request pairs issued by this processor. 
° FSB_RW_issue (bit 53): Counts back-to-back read-write transaction request pairs issued by this processor. 
¢ FSB_other_DBSY (bit 54): Count DBSY assertions by another agent (without a concurrent DRDY). 

¢ FSB_other_DRDY (bit 55): Count DRDY assertions by another agent. 

¢ FSB_other_snoop_stall (bit 56): Count snoop stalls on the FSB due to another agent. 

¢ FSB_other_BNR (bit 57): Count BNR assertions from another agent. 


18.6.7.5 Common Event Control Interface 


The MSR_EMON_L3_GL_CTL MSR provides simplified access to query overflow status of the GBSQ, GSNPQ, FSB 
event counters. It also provides control bit fields to freeze, unfreeze, or reset those counters. The following bit 
fields are supported: 


° GL_freeze_cmd (bit 0): Freeze the event counters specified by the GL_event_select field. 
¢ GL_unfreeze_cmd (bit 1): Unfreeze the event counters specified by the GL_event_select field. 


° GL_reset_cmd (bit 2): Clear the event count field of the event counters specified by the GL_event_select field. 
The event select field is not affected. 


° GL_event_select (bit 23:16): Selects one or more event counters to subject to specified command operations 
indicated by bits 2:0. Bit 16 corresponds to MSR_EMON_L3_CTR_CTLO, bit 23 corresponds to 
MSR_EMON_L3_CTR_CTL7. 


° GL_event_status (bit 55:48): Indicates the overflow status of each event counters. Bit 48 corresponds to 
MSR_EMON_L3_CTR_CTLO, bit 55 corresponds to MSR_EMON_L3_CTR_CTL/7. 


In the event control field (bits 63:32) of each MSR, if the saturate control (bit 59, see Figure 18-58 for example) is 
set, the event logic forces the value FFFF_FFFFH into the event count field instead of incrementing it. 


18.6.8 | Performance Monitoring (P6 Family Processor) 


The P6 family processors provide two 40-bit performance counters, allowing two types of events to be monitored 
simultaneously. These can either count events or measure duration. When counting events, a counter increments 
each time a specified event takes place or a specified number of events takes place. When measuring duration, it 
counts the number of processor clocks that occur while a specified condition is true. The counters can count events 
or measure durations that occur at any privilege level. 


18-130 Vol. 3B 


PERFORMANCE MONITORING 


NOTE 


The performance-monitoring events found at https://perfmon-events.intel.com/ are intended to 
be used as guides for performance tuning. Counter values reported are not guaranteed to be 
accurate and should be used as a relative guide for tuning. Known discrepancies are documented 
where applicable. 


The performance-monitoring counters are supported by four MSRs: the performance event select MSRs 
(PerfEvtSelO and PerfEvtSel1) and the performance counter MSRs (PerfCtrO and PerfCtri). These registers can be 
read from and written to using the RDMSR and WRMSR instructions, respectively. They can be accessed using 
these instructions only when operating at privilege level 0. The PerfCtrO and PerfCtri MSRs can be read from any 
privilege level using the RDPMC (read performance-monitoring counters) instruction. 


NOTE 


The PerfEvtSel0, PerfEvtSeli, PerfCtrO, and PerfCtr1 MSRs and the events listed for P6 family 
processors are model-specific for P6 family processors. They are not guaranteed to be available in 
other IA-32 processors. 


18.6.8.1 Perf€vtSel0 and Perf€vtSel1 MSRs 


The PerfEvtSel0 and PerfEvtSel1 MSRs control the operation of the performance-monitoring counters, with one 
register used to set up each counter. They specify the events to be counted, how they should be counted, and the 
privilege levels at which counting should take place. Figure 18-61 shows the flags and fields in these MSRs. 


The functions of the flags and fields in the PerfEvtSel0 and PerfEvtSeli MSRs are as follows: 


¢ Event select field (bits 0 through 7) — Selects the event logic unit to detect certain microarchitectural 
conditions. 


¢ Unit mask (UMASK) field (bits 8 through 15) — Further qualifies the event logic unit selected in the event 
select field to detect a specific microarchitectural condition. For example, for some cache events, the mask is 
used as a MESI-protocol qualifier of cache states. 


31 24.23 22 212019 1817 16 15 87 0 
Counter Mask | | a 
(CMASK) nN 4 DAG S| Unit Mask (UMASK) Event Select 


INV—Invert counter mask —— 
EN—Enable counters* 

INT—APIC interrupt enable ————— Reserved 
PC—Pin control 
E—Edge detect 


OS—Operating system mode ————____" * Only available in PerfEvtSel0. 
USR—User Mode 


Figure 18-61. Perf€vtSel0 and Perf€vtSell MSRs 


¢ USR (user mode) flag (bit 16) — Specifies that events are counted only when the processor is operating at 
privilege levels 1, 2 or 3. This flag can be used in conjunction with the OS flag. 


° OS (operating system mode) flag (bit 17) — Specifies that events are counted only when the processor is 
operating at privilege level 0. This flag can be used in conjunction with the USR flag. 


° E (edge detect) flag (bit 18) — Enables (when set) edge detection of events. The processor counts the 
number of deasserted to asserted transitions of any condition that can be expressed by the other fields. The 
mechanism is limited in that it does not permit back-to-back assertions to be distinguished. This mechanism 
allows software to measure not only the fraction of time spent in a particular state, but also the average length 
of time spent in such a state (for example, the time spent waiting for an interrupt to be serviced). 
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° PC (pin control) flag (bit 19) — When set, the processor toggles the PM/ pins and increments the counter 
when performance-monitoring events occur; when clear, the processor toggles the PM/ pins when the counter 
overflows. The toggling of a pin is defined as assertion of the pin for a single bus clock followed by deassertion. 


¢ INT (APIC interrupt enable) flag (bit 20) — When set, the processor generates an exception through its 
local APIC on counter overflow. 


¢ EN (Enable Counters) Flag (bit 22) — This flag is only present in the PerfEvtSel0 MSR. When set, 
performance counting is enabled in both performance-monitoring counters; when clear, both counters are 
disabled. 


° INV (invert) flag (bit 23) — When set, inverts the counter-mask (CMASK) comparison, so that both greater 
than or equal to and less than comparisons can be made (0: greater than or equal; 1: less than). Note if 
counter-mask is programmed to zero, INV flag is ignored. 


* Counter mask (CMASK) field (bits 24 through 31) — When nonzero, the processor compares this mask to 
the number of events counted during a single cycle. If the event count is greater than or equal to this mask, the 
counter is incremented by one. Otherwise the counter is not incremented. This mask can be used to count 
events only if multiple occurrences happen per clock (for example, two or more instructions retired per clock). 
If the counter-mask field is 0, then the counter is incremented each cycle by the number of events that 
occurred that cycle. 


18.6.8.2 PerfCtrO and PerfCtr1 MSRs 


The performance-counter MSRs (PerfCtrO and PerfCtr1) contain the event or duration counts for the selected 
events being counted. The RDPMC instruction can be used by programs or procedures running at any privilege level 
and in virtual-8086 mode to read these counters. The PCE flag in control register CR4 (bit 8) allows the use of this 
instruction to be restricted to only programs and procedures running at privilege level 0. 


The RDPMC instruction is not serializing or ordered with other instructions. Thus, it does not necessarily wait until 
all previous instructions have been executed before reading the counter. Similarly, subsequent instructions may 
begin execution before the RDPMC instruction operation is performed. 


Only the operating system, executing at privilege level 0, can directly manipulate the performance counters, using 
the RDMSR and WRMSR instructions. A secure operating system would clear the PCE flag during system initializa- 
tion to disable direct user access to the performance-monitoring counters, but provide a user-accessible program- 
ming interface that emulates the RDPMC instruction. 


The WRMSR instruction cannot arbitrarily write to the performance-monitoring counter MSRs (PerfCtrO and 
PerfCtr1). Instead, the lower-order 32 bits of each MSR may be written with any value, and the high-order 8 bits 
are sign-extended according to the value of bit 31. This operation allows writing both positive and negative values 
to the performance counters. 


18.6.8.3 Starting and Stopping the Performance-Monitoring Counters 


The performance-monitoring counters are started by writing valid setup information in the PerfEvtSel0 and/or 
PerfEvtSel1 MSRs and setting the enable counters flag in the PerfEvtSelO MSR. If the setup is valid, the counters 
begin counting following the execution of a WRMSR instruction that sets the enable counter flag. The counters can 
be stopped by clearing the enable counters flag or by clearing all the bits in the PerfEvtSelO and PerfEvtSeli MSRs. 
Counter 1 alone can be stopped by clearing the PerfEvtSel1 MSR. 


18.6.8.4 Event and Time-Stamp Monitoring Software 


To use the performance-monitoring counters and time-stamp counter, the operating system needs to provide an 
event-monitoring device driver. This driver should include procedures for handling the following operations: 


¢ Feature checking. 

¢ Initialize and start counters. 
¢ Stop counters. 

¢ Read the event counters. 

¢ Read the time-stamp counter. 
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The event monitor feature determination procedure must check whether the current processor supports the 
performance-monitoring counters and time-stamp counter. This procedure compares the family and model of the 
processor returned by the CPUID instruction with those of processors known to support performance monitoring. 
(The Pentium and P6 family processors support performance counters.) The procedure also checks the MSR and 
TSC flags returned to register EDX by the CPUID instruction to determine if the MSRs and the RDTSC instruction 
are supported. 


The initialize and start counters procedure sets the PerfEvtSel0 and/or PerfEvtSel1l MSRs for the events to be 
counted and the method used to count them and initializes the counter MSRs (PerfCtrO and PerfCtr1) to starting 
counts. The stop counters procedure stops the performance counters (see Section 18.6.8.3, “Starting and Stop- 
ping the Performance-Monitoring Counters”). 


The read counters procedure reads the values in the PerfCtrO and PerfCtri1 MSRs, and a read time-stamp counter 
procedure reads the time-stamp counter. These procedures would be provided in lieu of enabling the RDTSC and 
RDPMC instructions that allow application code to read the counters. 


18.6.8.5 Monitoring Counter Overflow 


The P6 family processors provide the option of generating a local APIC interrupt when a performance-monitoring 
counter overflows. This mechanism is enabled by setting the interrupt enable flag in either the PerfEvtSelO or the 
PerfEvtSeli MSR. The primary use of this option is for statistical performance sampling. 


To use this option, the operating system should do the following things on the processor for which performance 
events are required to be monitored: 


¢ Provide an interrupt vector for handling the counter-overflow interrupt. 

® Initialize the APIC PERF local vector entry to enable handling of performance-monitor counter overflow events. 

¢ Provide an entry in the IDT that points to a stub exception handler that returns without executing any instruc- 
tions. 

¢ Provide an event monitor driver that provides the actual interrupt handler and modifies the reserved IDT entry 
to point to its interrupt routine. 

When interrupted by a counter overflow, the interrupt handler needs to perform the following actions: 

¢ Save the instruction pointer (EIP register), code-segment selector, TSS segment selector, counter values and 
other relevant information at the time of the interrupt. 

¢ Reset the counter to its initial setting and return from the interrupt. 


An event monitor application utility or another application program can read the information collected for analysis 
of the performance of the profiled application. 


18.6.9 Performance Monitoring (Pentium Processors) 


The Pentium processor provides two 40-bit performance counters, which can be used to count events or measure 
duration. The counters are supported by three MSRs: the control and event select MSR (CESR) and the perfor- 
mance counter MSRs (CTRO and CTR1). These can be read from and written to using the RDMSR and WRMSR 
instructions, respectively. They can be accessed using these instructions only when operating at privilege level 0. 


Each counter has an associated external pin (PMO/BPO and PM1/BP1), which can be used to indicate the state of 
the counter to external hardware. 


NOTES 


The CESR, CTRO, and CTR1 MSRs and the events listed for Pentium processors are model-specific 
for the Pentium processor. 


The performance-monitoring events found at https://perfmon-events.intel.com/ are intended to 
be used as guides for performance tuning. Counter values reported are not guaranteed to be 
accurate and should be used as a relative guide for tuning. Known discrepancies are documented 
where applicable. 
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18.6.9.1 Control and Event Select Register (CESR) 


The 32-bit control and event select MSR (CESR) controls the operation of performance-monitoring counters CTRO 
and CTRi1 and the associated pins (see Figure 18-62). To control each counter, the CESR register contains a 6-bit 
event select field (ESO and ES1), a pin control flag (PCO and PC1), and a 3-bit counter control field (CCO and CC1). 
The functions of these fields are as follows: 


° ESO and ES1 (event select) fields (bits 0-5, bits 16-21) — Selects (by entering an event code in the field) 
up to two events to be monitored. 


31 262524 2221 1615 1098 65 0 
P P 
cl] cc ES1 c| cco ESO 
1 0 


PC1—Pin control 1 
CC1—Counter control 1 
ES1—Event select 1 
PCO—Pin control 0 
CCO—Counter control 0 
ESO—Event select 0 


Reserved 


Figure 18-62. CESR MSR (Pentium Processor Only) 


* CCO and CCi (counter control) fields (bits 6-8, bits 22-24) — Controls the operation of the counter. 
Control codes are as follows: 


000 — Count nothing (counter disabled). 

001 — Count the selected event while CPL is 0, 1, or 2. 
010 — Count the selected event while CPL is 3. 

011 — Count the selected event regardless of CPL. 
100 — Count nothing (counter disabled). 

101 — Count clocks (duration) while CPL is 0, 1, or 2. 
110 — Count clocks (duration) while CPL is 3. 

111 — Count clocks (duration) regardless of CPL. 


The highest order bit selects between counting events and counting clocks (duration); the middle bit enables 
counting when the CPL is 3; and the low-order bit enables counting when the CPL is O, 1, or 2. 


¢ PCO and PC1 (pin control) flags (bits 9, 25) — Selects the function of the external performance-monitoring 
counter pin (PMO/BPO and PM1/BP1). Setting one of these flags to 1 causes the processor to assert its 
associated pin when the counter has overflowed; setting the flag to 0 causes the pin to be asserted when the 
counter has been incremented. These flags permit the pins to be individually programmed to indicate the 
overflow or incremented condition. The external signaling of the event on the pins will lag the internal event by 
a few clocks as the signals are latched and buffered. 


While a counter need not be stopped to sample its contents, it must be stopped and cleared or preset before 
switching to a new event. It is not possible to set one counter separately. If only one event needs to be changed, 
the CESR register must be read, the appropriate bits modified, and all bits must then be written back to CESR. At 
reset, all bits in the CESR register are cleared. 


18.6.9.2 Use of the Performance-Monitoring Pins 


When performance-monitor pins PMO/BPO and/or PM1/BP1 are configured to indicate when the performance- 
monitor counter has incremented and an “occurrence event” is being counted, the associated pin is asserted (high) 
each time the event occurs. When a “duration event” is being counted, the associated PM pin is asserted for the 
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entire duration of the event. When the performance-monitor pins are configured to indicate when the counter has 
overflowed, the associated PM pin is asserted when the counter has overflowed. 


When the PMO/BPO and/or PM1i/BP1 pins are configured to signal that a counter has incremented, it should be 
noted that although the counters may increment by 1 or 2 in a single clock, the pins can only indicate that the event 
occurred. Moreover, since the internal clock frequency may be higher than the external clock frequency, a single 
external clock may correspond to multiple internal clocks. 


A “count up to” function may be provided when the event pin is programmed to signal an overflow of the counter. 
Because the counters are 40 bits, a carry out of bit 39 indicates an overflow. A counter may be preset to a specific 
value less then 27° — 1. After the counter has been enabled and the prescribed number of events has transpired, 
the counter will overflow. 


Approximately 5 clocks later, the overflow is indicated externally and appropriate action, such as signaling an inter- 
rupt, may then be taken. 


The PMO/BPO and PM1/BP1 pins also serve to indicate breakpoint matches during in-circuit emulation, during which 
time the counter increment or overflow function of these pins is not available. After RESET, the PMO/BPO and 
PM1/BP1 pins are configured for performance monitoring, however a hardware debugger may reconfigure these 
pins to indicate breakpoint matches. 


18.6.9.3 Events Counted 


Events that performance-monitoring counters can be set to count and record (using CTRO and CTR1) are divided in 
two categories: occurrence and duration: 


¢ Occurrence events — Counts are incremented each time an event takes place. If PMO/BPO or PM1/BP1 pins 
are used to indicate when a counter increments, the pins are asserted each clock counters increment. But if an 
event happens twice in one clock, the counter increments by 2 (the pins are asserted only once). 


¢ Duration events — Counters increment the total number of clocks that the condition is true. When used to 
indicate when counters increment, PMO/BPO and/or PM1/BP1 pins are asserted for the duration. 


18.7. COUNTING CLOCKS 


The count of cycles, also known as clockticks, forms the basis for measuring how long a program takes to execute. 
Clockticks are also used as part of efficiency ratios like cycles per instruction (CPI). Processor clocks may stop 
ticking under circumstances like the following: 


¢ The processor is halted when there is nothing for the CPU to do. For example, the processor may halt to save 
power while the computer is servicing an I/O request. When Intel Hyper-Threading Technology is enabled, both 
logical processors must be halted for performance-monitoring counters to be powered down. 


¢ The processor is asleep as a result of being halted or because of a power-management scheme. There are 
different levels of sleep. In the some deep sleep levels, the time-stamp counter stops counting. 


In addition, processor core clocks may undergo transitions at different ratios relative to the processor’s bus clock 
frequency. Some of the situations that can cause processor core clock to undergo frequency transitions include: 


¢ TM2 transitions. 
¢ Enhanced Intel SpeedStep Technology transitions (P-state transitions). 


For Intel processors that support TM2, the processor core clocks may operate at a frequency that differs from the 
Processor Base frequency (as indicated by processor frequency information reported by CPUID instruction). See 
Section 18.7.2 for more detail. 


Due to the above considerations there are several important clocks referenced in this manual: 


° Base Clock — The frequency of this clock is the frequency of the processor when the processor is not in turbo 
mode, and not being throttled via Intel SpeedStep. 


¢ Maximum Clock — This is the maximum frequency of the processor when turbo mode is at the highest point. 
¢ Bus Clock — These clockticks increment at a fixed frequency and help coordinate the bus on some systems. 
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° Core Crystal Clock — This is a clock that runs at fixed frequency; it coordinates the clocks on all packages 
across the system. 


¢ Non-halted Clockticks — Measures clock cycles in which the specified logical processor is not halted and is 
not in any power-saving state. When Intel Hyper-Threading Technology is enabled, ticks can be measured on a 
per-logical-processor basis. There are also performance events on dual-core processors that measure 
clockticks per logical processor when the processor is not halted. 


¢ Non-sleep Clockticks — Measures clock cycles in which the specified physical processor is not in a sleep mode 
or in a power-saving state. These ticks cannot be measured on a logical-processor basis. 


¢ Time-stamp Counter — See Section 17.17, “Time-Stamp Counter”. 


° Reference Clockticks — TM2 or Enhanced Intel SpeedStep technology are two examples of processor 
features that can cause processor core clockticks to represent non-uniform tick intervals due to change of bus 
ratios. Performance events that counts clockticks of a constant reference frequency was introduced Intel Core 
Duo and Intel Core Solo processors. The mechanism is further enhanced on processors based on Intel Core 
microarchitecture. 


Some processor models permit clock cycles to be measured when the physical processor is not in deep sleep (by 
using the time-stamp counter and the RDTSC instruction). Note that such ticks cannot be measured on a per- 
logical-processor basis. See Section 17.17, “Time-Stamp Counter,” for detail on processor capabilities. 


The first two methods use performance counters and can be set up to cause an interrupt upon overflow (for 
sampling). They may also be useful where it is easier for a tool to read a performance counter than to use a time 
stamp counter (the timestamp counter is accessed using the RDTSC instruction). 


For applications with a significant amount of I/O, there are two ratios of interest: 


¢ Non-halted CPI — Non-halted clockticks/instructions retired measures the CPI for phases where the CPU was 
being used. This ratio can be measured on a logical-processor basis when Intel Hyper-Threading Technology is 
enabled. 


¢ Nominal CPI — Time-stamp counter ticks/instructions retired measures the CPI over the duration of a 
program, including those periods when the machine halts while waiting for I/O. 


18.7.1 Non-Halted Reference Clockticks 


Software can use UnHalted Reference Cycles on either a general purpose performance counter using event mask 
Ox3C and umask 0x01 or on fixed function performance counter 2 to count at a constant rate. These events count 
at a consistent rate irrespective of P-state, TM2, or frequency transitions that may occur to the processor. The 
UnHalted Reference Cycles event may count differently on the general purpose event and fixed counter. 


18.7.2 | Cycle Counting and Opportunistic Processor Operation 


As a result of the state transitions due to opportunistic processor performance operation (see Chapter 14, “Power 
and Thermal Management”), a logical processor or a processor core can operate at frequency different from the 
Processor Base frequency. 


The following items are expected to hold true irrespective of when opportunistic processor operation causes state 
transitions: 


¢ The time stamp counter operates at a fixed-rate frequency of the processor. 


¢ The IA32_MPERF counter increments at a fixed frequency irrespective of any transitions caused by opportu- 
nistic processor operation. 


° The IA32_FIXED_CTR2 counter increments at the same TSC frequency irrespective of any transitions caused by 
opportunistic processor operation. 


¢ The Local APIC timer operation is unaffected by opportunistic processor operation. 


¢ The TSC, IA32_MPERF, and IA32_FIXED_CTR2 operate at close to the maximum non-turbo frequency, which is 
equal to the product of scalable bus frequency and maximum non-turbo ratio. 
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18.7.3. Determining the Processor Base Frequency 


For Intel processors in which the nominal core crystal clock frequency is enumerated in CPUID.15H.ECX and the 
core crystal clock ratio is encoded in CPUID.15H (see Table 3-8 “Information Returned by CPUID Instruction”), the 
nominal TSC frequency can be determined by using the following equation: 


Nominal TSC frequency = ( CPUID.15H.ECX[31:0] * CPUID.15H.EBX[3 1:0] ) + CPUID.15H.EAX[31:0] 


For Intel processors in which CPUID.15H.EBX[31:0] + CPUID.0x15.EAX[31:0] is enumerated but CPUID.15H.ECX 
is not enumerated, Table 18-85 can be used to look up the nominal core crystal clock frequency. 


Table 18-85. Nominal Core Crystal Clock Frequency 


Processor Families/Processor Number Series! Nominal Core Crystal Clock Frequency 


Intel® Xeon® Processor Scalable Family with CPUID signature O6_55H. 25 MHz 


6th and 7th generation Intel® Core™ processors and Intel® Xeon® W Processor Family. | 24 MHz 


Next Generation Intel® Atom™ processors based on Goldmont Microarchitecture with 19.2 MHz 
CPUID signature 06_5CH (does not include Intel Xeon processors). 


NOTES: 
1. For any processor in which CPUID.15H is enumerated and MSR_PLATFORM_INFO[1 5:8] (which gives the scalable bus frequency) is 
available, a more accurate frequency can be obtained by using CPUID.15H. 


18.7.3.1 For Intel® Processors Based on Microarchitecture Code Name Sandy Bridge, Ivy Bridge, 
Haswell and Broadwell 


The scalable bus frequency is encoded in the bit field MSR_PLATFORM_INFO[15:8] and the nominal TSC frequency 
can be determined by multiplying this number by a bus speed of 100 MHz. 


18.7.3.2 For Intel® Processors Based on Microarchitecture Code Name Nehalem 


The scalable bus frequency is encoded in the bit field MSR_PLATFORM_INFO[15:8] and the nominal TSC frequency 
can be determined by multiplying this number by a bus speed of 133.33 MHz. 


18.7.3.3 For Intel® Atom™ Processors Based on the Silvermont Microarchitecture (Including Intel 
Processors Based on Airmont Microarchitecture) 


The scalable bus frequency is encoded in the bit field MSR_PLATFORM_INFO[15:8] and the nominal TSC frequency 
can be determined by multiplying this number by the scalable bus frequency. The scalable bus frequency is 
encoded in the bit field MSR_FSB_FREQ[2:0] for Intel Atom processors based on the Silvermont microarchitecture, 
and in bit field MSR_FSB_FREQ[3:0] for processors based on the Airmont microarchitecture; see Chapter 2, 
“Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
4. 
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18.7.3.4 For Intel® Core™ 2 Processor Family and for Intel* Xeon® Processors Based on Intel Core 
Microarchitecture 


For processors based on Intel Core microarchitecture, the scalable bus frequency is encoded in the bit field 
MSR_FSB_FREQ[2:0] at (OCDH), see Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 4. The maximum resolved bus ratio can be read from the 
following bit field: 


¢ If XE operation is disabled, the maximum resolved bus ratio can be read in MSR_PLATFORM_ID[12:8]. It 
corresponds to the Processor Base frequency. 


¢ IF XE operation is enabled, the maximum resolved bus ratio is given in MSR_PERF_STATUS[44:40], it 
corresponds to the maximum XE operation frequency configured by BIOS. 


XE operation of an Intel 64 processor is implementation specific. XE operation can be enabled only by BIOS. If 
MSR_PERF_STATUS[31] is set, XE operation is enabled. The MSR_PERF_STATUS[31] field is read-only. 


18.8 IA32_PERF_CAPABILITIES MSR ENUMERATION 


The layout of IA32_PERF_CAPABILITIES MSR is shown in Figure 18-63; it provides enumeration of a variety of 
interfaces: 


e JTA32_PERF_CAPABILITIES.LBR_FMT[bits 5:0]: encodes the LBR format, details are described in Section 
17.4.8.1. 


e = JTA32_PERF_CAPABILITIES.PEBSTrap[6]: Trap/Fault-like indicator of PEBS recording assist; see Section 
18.6.2.4.2. 


e = =TA32_PERF_CAPABILITIES.PEBSArchRegs[7]: Indicator of PEBS assist save architectural registers; see Section 
18.6.2.4.2. 


¢ TA32_PERF_CAPABILITIES.PEBS_FMT[bits 11:8]: Specifies the encoding of the layout of PEBS records; see 
Section 18.6.2.4.2. 


e 1A32_PERF_CAPABILITIES.FREEZE_WHILE_SMM[12]: Indicates IA32_DEBUGCTL.FREEZE_WHILE_SMM is 
supported if 1, see Section 18.8.1. 


e = JTA32_PERF_CAPABILITIES.FULL_WRITE[13]: Indicates the processor supports IA32_A_PMCx interface for 
updating bits 32 and above of IA32_PMCx; see Section 18.2.6. 


e = TA32_PERF_CAPABILITIES.PEBS_BASELINE [bit 14]: If set, the following is true: 


— The IA32_PEBS_ENABLE MSR (address 3F1H) exists and all architecturally enumerated fixed and general- 
purpose counters have corresponding bits in IA32_PEBS_ENABLE that enable generation of PEBS records. 
The general-purpose counter bits start at bit IA32_PEBS_ENABLE[0], and the fixed counter bits start at bit 
IA32_PEBS_ENABLE[32]. 


— The format of the PEBS record is enumerated by IA32_PERF_CAPABILITIES.PEBS_FMT; see Section 
18.6.2.4.2. 


— Extended PEBS is supported. All counters support the PEBS facility, and all events (both precise and non- 
precise) can generate PEBS records when PEBS is enabled for that counter. Note that not all events may be 
available on all counters. 


— Adaptive PEBS is supported. The PEBS_DATA_CFG MSR (address 3F2H) and adaptive record enable bits 
(IA32_PERFEVTSELx.Adaptive_Record and IA32_FIXED_CTR_CTRL.FCx_Adaptive_Record) are supported. 
The definition of the PEBS_ DATA_CFG MSR, including which bits are supported and how they affect the 
record, is enumerated by IA32_PERF_CAPABILITIES.PEBS_FMT; see Section 18.9.2.3. 


e =JTA32_PERF_CAPABILITIES.PERF_METRICS_AVAILABLE[15]: If set, indicates that the architecture provides 
built in support for TMA Li metrics through the PERF_METRICS MSR, see Section 18.3.9.3. 


¢ = JTA32_PERF_CAPABILITIES.PEBS_OUTPUT_PT_AVAIL[16]: If set on parts that enumerate support for Intel PT 
(CPUID.0x7.0.EBX[25]=1), setting IA32_PEBS_ENABLE.PEBS_OUTPUT to 01B will result in PEBS output being 
written into the Intel PT trace stream. See Section 18.5.5.2. 
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63 1615 1312 11 8 76 543210 


PEBS_OUTPUT_PT_AVAIL (R/O) 

PERF_METRICS_AVAILABLE (R/O) 
FW_WRITE (R/O) 
SMM_FREEZE (R/O) 
PEBS_REC_FMT (R/O) 
PEBS ARCH_REG (R/O) 
PEBS_TRAP (R/O) 
LBR_FMT (R/O) 


Reserved 


Figure 18-63. Layout of |A32_PERF_CAPABILITIES MSR 


18.8.1 Filtering of SMM Handler Overhead 


When performance monitoring facilities and/or branch profiling facilities (see Section 17.5, “Last Branch, Interrupt, 
and Exception Recording (Intel® Core™ 2 Duo and Intel® Atom™ Processors)”) are enabled, these facilities 
capture event counts, branch records and branch trace messages occurring in a logical processor. The occurrence 
of interrupts, instruction streams due to various interrupt handlers all contribute to the results recorded by these 
facilities. 


If CPUID.01H:ECX.PDCM[bit 15] is 1, the processor supports the IA32_PERF_CAPABILITIES MSR. If 
IA32_PERF_CAPABILITIES.FREEZE_WHILE_SMM[Bit 12] is 1, the processor supports the ability for system soft- 
ware using performance monitoring and/or branch profiling facilities to filter out the effects of servicing system 
management interrupts. 


If the FREEZE_WHILE_SMM capability is enabled on a logical processor and after an SMI is delivered, the processor 
will clear all the enable bits of IA32_PERF_GLOBAL_CTRL, save a copy of the content of IA32_DEBUGCTL and 
disable LBR, BTF, TR, and BTS fields of IA32_DEBUGCTL before transferring control to the SMI handler. 


The enable bits of IA32_PERF_GLOBAL_CTRL will be set to 1, the saved copy of IA32_DEBUGCTL prior to SMI 
delivery will be restored , after the SMI handler issues RSM to complete its servicing. 


It is the responsibility of the SMM code to ensure the state of the performance monitoring and branch profiling facil- 
ities are preserved upon entry or until prior to exiting the SMM. If any of this state is modified due to actions by the 
SMM code, the SMM code is required to restore such state to the values present at entry to the SMM handler. 


System software is allowed to set IA32_DEBUGCTL.FREEZE_WHILE_SMM[bit 14] to 1 only supported as indicated 
by IA32_PERF_CAPABILITIES.FREEZE_WHILE_SMM[Bit 12] reporting 1. 


18.9 PEBS FACILITY 


18.9.1 Extended PEBS 


° The Extended PEBS feature supports Processor Event Based Sampling (PEBS) on all counters, both fixed 
function and general purpose; and all performance monitoring events, both precise and non-precise. PEBS can 
be enabled for the general purpose counters using PEBS_EN_PMCi bits of IA32_PEBS_ENABLE (i = 0, 1,..n). 
PEBS can be enabled for 'i' fixed function counters using the PEBS_EN_FIXEDi bits of IA32_PEBS_ENABLE (i = 
0,1, ...m). 
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63 n 32 31 m 10 


PEBS_EN_FIXEDn(RMW) _| 


PEBS_EN_FIXED1 (RW) 
PEBS _EN_FIXEDO (R/W) 


PEBS_EN_PMCm (R/W) 
PEBS_EN_PMC1 (RW) 
PEBS EN_PMCO (R/W) 


Reserved RESET Value — 00000000 _ 00000000 H 


Figure 18-64. Layout of IA32_PEBS_ENABLE MSR 


A PEBS record due to a precise event will be generated after an instruction that causes the event when the counter 
has already overflowed. A PEBS record due to a non-precise event will occur at the next opportunity after the 
counter has overflowed, including immediately after an overflow is set by an MSR write. 


Currently, IA32_FIXED_CTRO counts instructions retired and is a precise event. IA32_FIXED_CTR1, 
IA32_FIXED_CTR2 ... IA32_FIXED_CTRm count as non-precise events. 


The Applicable Counter field in the Basic Info Group of the PEBS record indicates which counters caused the PEBS 
record to be generated. It is in the same format as the enable bits for each counter in IA32_PEBS_ENABLE. As an 
example, an Applicable Counter field with bits 2 and 32 set would indicate that both general purpose counter 2 and 
fixed function counter 0 generated the PEBS record. 


° To properly use PEBS for the additional counters, software will need to set up the counter reset values in PEBS 
portion of the DS_BUFFER_MANAGEMENT_AREA data structure that is indicated by the IA32_DS_AREA 
register. The layout of the DS_BUFFER_MANAGEMENT_AREA is shown in Figure 18-65. When a counter 
generates a PEBS records, the appropriate counter reset values will be loaded into that counter. In the above 
example where general purpose counter 2 and fixed function counter 0 generated the PEBS record, general 
purpose counter 2 would be reloaded with the value contained in PEBS GP Counter 2 Reset (offset 50H) and 
fixed function counter 0 would be reloaded with the value contained in PEBS Fixed Counter 0 Reset (offset 
80H). 
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|A32_DS_AREA MSR 


DS Buffer Management BTS Buffer 
63 BTS BufferBase 0 |*00H 5! 
Branch Record 0 
BTS Index 08H 
BTS Absolute Maximum 10H 
Branch Record 1 
BTS Interrupt Threshold 18H 
PEBS Buffer Base 20H > 
PEBS Index 28H 
PEBS Absolute Maximum 30H + 
PEBS Interrupt Threshold 38H Branch Record N 
PEBS GP Counter 0 Reset 40H , 
PEBS GP Counter 1 Reset ra 
m 
w 
7) 
ve 9 PEBS Buffer 
= 
PEBS GP Counter m Reset a 7 
By PEBS Record 0 
PEBS Fixed Counter 0 Reset 80H a 
oe? PEBS Record 1 
PEBS Fixed Counter n Reset > 


PEBS Record N 


Figure 18-65. PEBS Programming Environment 


Extended PEBS support debuts on Intel® Atom processors based on the Goldmont Plus microarchitecture and 
future Intel® Core™ processors based on the Ice Lake microarchitecture. 


18.9.2 Adaptive PEBS 

The PEBS facility has been enhanced to collect the following CPU state in addition to GPRs, EventingIP, TSC and 
memory access related information collected by legacy PEBS: 

¢ XMM registers 

¢ LBR records (TO/FROM/INFO) 

The PEBS record is restructured where fields are grouped into Basic group, Memory group, GPR group, XMM group 
and LBR group. A new register MSR_PEBS_DATA_CFG provides software the capability to select data groups of 
interest and thus reduce the record size in memory and record generation latency. Hence, a PEBS record's size and 


layout vary based on the selected groups. The MSR also allows software to select LBR depth for branch data 
records. 


By default, the PEBS record will only contain the Basic group. Optionally, each counter can be configured to 
generate a PEBS records with the groups specified in MSR_PEBS_DATA_CFG. 
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Details and examples for the Adaptive PEBS capability follow below. 


18.9.2.1 Adaptive_Record Counter Control 


e =TA32_PERFEVTSELx.Adaptive_Record[34]: If this bit is set and IA32_PEBS_ENABLE.PEBS_EN_PMCx is set for 
the corresponding GP counter, an overflow of PMCx results in generation of an adaptive PEBS record with state 
information based on the selections made in MSR_PEBS_DATA_CFG. If this bit is not set, a basic record is 
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1A32_PerfEvtSelX 
Address Range: 0x186 to 0x18D 
Scope: Thread 
Reset value: 0x00000000.0000000 


Figure 18-66. Layout of IA32_Perf€vtSelX MSR Supporting Adaptive PEBS 


e = TA32_FIXED_CTR_CTRL.FCx_Adaptive_Record: If this bit is set and IA32_PEBS_ENABLE.PEBS_EN_FIXEDx is 
set for the corresponding Fixed counter, an overflow of FixedCtrx results in generation of an adaptive PEBS 
record with state information based on the selections made in MSR_PEBS_DATA_CFG. If this bit is not set, a 
basic record is generated. 
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Reserved 


AnyThr_FC2 
AnyThr_FC1 
AnyThr_FCO 


Deprecated 


PMI_FC3 
EN_FC3 
PMI_FC2 
EN_FC2 
EN_FC1 
EN_FCO 


Reserved 
PMI_FC1 
PMI_FCO 


31 23 15 7 0 


63 55 47 39 


w 
Nh 


FC3_ADAPTIVE_RECORD 


FC2_ADAPTIVE_RECORD 
FC1_ADAPTIVE_RECORD 


FCO_ADAPTIVE_RECORD 


IA32_Fixed_CTR_CTRL 
Address: 38DH 
Scope: Thread 
Reset value: 0x00000000.00000000 


Figure 18-67. Layout of IA32_FIXED_CTR_CTRL MSR Supporting Adaptive PEBS 


18.9.2.2 PEBS Record Format 


The data fields in the PEBS record are aggregated into five groups which are described in the sub-sections below. 
Processors that support Adaptive PEBS implement a new MSR called MSR_PEBS_DATA_CFG which allows software 
to select the data groups to be captured. The data groups are not placed at fixed locations in the PEBS record, but 
are positioned immediately after one another, thus making the record format/size variable based on the groups 
selected. 


18.9.2.2.1 Basic Info 


The Basic group contains essential information for software to parse a record along with several critical fields. It is 
always collected. 


Table 18-86. Basic Info Group 


Field Name Bit Width Description 


Record Format [47:0] This field indicates which data groups are included in the record. The field is zero if 
none of the counters that triggered the current PEBS record have their 
Adaptive_Record bit set. Otherwise it contains the value of MSR_PEBS_DATA_CFG. 


[63:48] This field provides the size of the current record in bytes. Selected groups are 
packed back-to-back in the record without gaps or padding for unselected groups. 
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Table 18-86. Basic Info Group (Contd.) 


Instruction Pointer [63:0] This field reports the Eventing Instruction Pointer (EventingIP) of the retired 
instruction that triggered the PEBS record generation. Note that this field is 
different than R/EIP which records the instruction pointer of the next instruction 
to be executed after record generation. The legacy R/EIP field has been removed. 


Applicable Counters [63:0] The Applicable Counters field indicates which counters triggered the generation of 
the PEBS record, linking the record to specific events. This allows software to 
correlate the PEBS record entry properly with the instruction that caused the 
event, even when multiple counters are configured to generate PEBS records and 
multiple bits are set in the field. 


TSC [63:0] This field provides the time stamp counter value when the PEBS record was 
generated. 


18.9.2.2.2 Memory Access Info 


This group contains the legacy PEBS memory-related fields; see Section 18.3.1.1.2. 


Table 18-87. Memory Access Info Group 


Field Name Bit Width Description 


Memory Access Address [63:0] This field contains the linear address of the source of the load, or linear address of 
the destination (target) of the store. This value is written as a 64-bit address in 
canonical form. 


Memory Auxiliary Info [63:0] When MEM_TRANS_RETIRED.* event is configured in a General Purpose counter, 
this field contains an encoded value indicating the memory hierarchy source which 
satisfied the load. These encodings are detailed in Table 18-4 and Table 18-13. If 
the PEBS assist was triggered for a store uop, this field will contain information 
indicating the status of the store, as detailed in Table 18-14. 


Memory Access Latency [63:0] When MEM_TRANS_RETIRED.* event is configured in a General Purpose counter, 
this field contains the latency to service the load in core clock cycles. 


TSX Auxiliary Info [31:0] This field contains the number of cycles in the last TSX region, regardless of 
whether that region had aborted or committed. 


[63:31] This field contains the abort details. Refer to Section 18.3.6.5.1. 


18.9.2.2.3 GPRs 


This group is captured when the GPR bit is enabled in MSR_PEBS_DATA_CFG. GPRs are always 64 bits wide. If they 
are selected for non 64-bit mode, the upper 32-bit of the legacy RAX - RDI and all contents of R8-15 GPRs will be 
filled with Os. In 64bit mode, the full 64 bit value of each register is written. 


The order differs from legacy. The table below shows the order of the GPRs in Ice Lake microarchitecture. 


Table 18-88. GPRs in Ice Lake Microarchitecture 


Field Name Bit Width 
RFLAGS [63:0] 
RIP [63:0] 
RAX [63:0] 
RCX* [63:0] 
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Table 18-88. GPRs in Ice Lake Microarchitecture (Contd.) 


RDX* [63:0] 
RBX* [63:0] 
RSP* [63:0] 
RBP* [63:0] 
RSI* [63:0] 
RDI* [63:0] 
R8 [63:0] 
R15 [63:0] 


The machine state reported in the PEBS record is the committed machine state immediately after the instruction 
that triggers PEBS completes. 


For instance, consider the following instruction sequence: 
MOV eax, [eax]; triggers PEBS record generation 
NOP 


If the mov instruction triggers PEBS record generation, the EventingIP field in the PEBS record will report the 
address of the mov, and the value of EAX in the PEBS record will show the value read from memory, not the target 
address of the read operation. And the value of RIP will contain the linear address of the nop. 


18.9.2.2.4 XMMs 


This group is captured when the XMM bit is enabled in MSR_PEBS_DATA_CFG and SSE is enabled. If SSE is not 
enabled, the fields will contain zeroes. XMM8-XMM15 will also contain zeroes if not in 64-bit mode. 


Table 18-89. XMMs 


Field Name Bit Width 


XMMO [127:0] 


XMM15 [127:0] 


18.9.2.2.5 LBRs 


To capture LBR data in the PEBS record, the LBR bit in MSR_PEBS_DATA_CFG must be enabled. The number of LBR 
entries included in the record can be configured in the LBR_entries field of MSR_PEBS_DATA_CFG. 
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Table 18-90. LBRs 


Field Name Bit Width Description 
LBR[].FROM [63:0] Branch from address. 
LBR[].TO [63:0] Branch to address. 
LBR[].INFO [63:0] Other LBR information, like timing. This field is described in more 


detail in Section 17.12.1, "MSR_LBR_INFO_x MSR". 


LBR entries are recorded into the record starting at LBR[TOS] and proceeding to LBR[TOS-1] and following. Note 
that LBR index is modulo the number of LBRs supporting on the processor. 


18.9.2.3  MSR_PEBS_DATA_CFG 


Bits in MSR_PEBS_DATA_CFG can be set to include data field blocks/groups into adaptive records. The Basic Info 
group is always included in the record. Additionally, the number of LBR entries included in the record is configu- 


rable. 


Reserved 


LBR Entries 


LBRs 

XMMs 

GPRs 
Memory Info 


31 


23 


31 


63 


55 


47 


MSR_PEBS_DATA_CFG 
Address: 3F2H 
Scope: Thread 
Reset value : 0x00000000 .00000000 


39 


63 
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Table 18-91. MSR_PEBS_CFG Programming! 


Bit Bit Index Access Description 

Memory Info 0 R/W Setting this bit will capture memory information such as the linear address, 
data source and latency of the memory access in the PEBS record. 

GPRs 1 R/W Setting this bit will capture the contents of the General Purpose registers 
in the PEBS record. 

XMMs 2 R/W Setting this bit will capture the contents of the XMM registers in the PEBS 
record. 

LBRs 3 R/W Setting this bit will capture LBR TO, FROM and INFO in the PEBS record. 

Reserved? 23:4 NA Reserved 

LBR Entries 31:24 R/W Set the field to the desired number of entries minus 1. For example, if the 
LBR_entries field is 0, a single entry will be included in the record. To 
include 32 LBR entries, set the LBR_entries field to 31 (Ox1F). To ensure 
all PEBS records are 16-byte aligned, it is recommended to select an even 
number of LBR entries (programmed into LBR_entries as an odd number). 

NOTES: 


1. A write to the MSR will be ignored when IA32_MISC_ENABLE.PERFMON_AVAILABLE is zero (default). 
2. Writing to the reserved bits will cause a GP fault. 


18.9.2.4 PEBS Record Examples 


The following example shows the layout of the PEBS record when all data groups are selected (all valid bits in 
MSR_PEBS_DATA_CFG are set) and maximum number of LBRs are selected. There are no gaps in the PEBS record 
when a subset of the groups are selected, thus keeping the layout compact. Implementations that do not support 
some features will have to pad zeroes in the corresponding fields. 


Table 18-92. PEBS Record Example 1 


Offset Group Name Field Name Legacy Name (If Different) 
0x0 Basic Info Record Format New 
Record Size New 
0x8 Instruction Pointer EventingRIP 
0x10 Applicable Counters 
0x18 TSC 
0x20 Memory Info Memory Access Address DLA 
0x28 Memory Auxiliary Info DATA_SRC 
0x30 Memory Access Latency Load Latency 
0x38 TSX Auxiliary Info HLE Information 
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Table 18-92. PEBS Record Example 1 


0x40 GPRs RFLAGS 

0x48 RIP 

0x50 RAX 

0x88 RDI 

0x90 R8 

OxC8 R15 

OxDO XMMs XMMO New 
0x1CO XMM15 

0x1D0 LBRs LBR[TOS].FROM New 
0x1D8 LBR[TOS].TO 

0x1€0 LBR[TOS].INFO 

0x4B8 LBR[TOS +1].FROM 

0x4CO LBR[TOS +1].TO 

0x4C8 LBR[TOS +1].INFO 


The following example shows the layout of the PEBS record when Basic, GPR, and LBR group with 3 LBR entries are 
selected. 


Table 18-93. PEBS Record Example 2 


Offset Group Name Field Name Legacy Name (If Different) 
0x0 Basic Info Record Format New 
Record Size New 
0x8 Instruction Pointer EventingRIP 
0x10 Applicable Counters 
0x18 TSC 
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Table 18-93. PEBS Record Example 2 


0x20 GPRs RFLAGS 

0x28 RIP 

0x30 RAX 

0x68 RDI 

0x70 R8 

OxA8 R15 

0xBO LBRs LBR[TOS].FROM New 
OxB8 LBR[TOS].TO 

OxCO LBR[TOS].INFO 
OxE0 LBR[TOS +1].FROM 
OxE8 LBR[TOS +1].TO 
OxFO LBR[TOS +1].INFO 


18.9.3 Precise Distribution of Instructions Retired (PDIR) Facility 


Precise Distribution of Instructions Retired Facility is available via PEBS on some microarchitectures. Refer to 
Section 18.3.4.4.4. Counters that support PDIR also vary. See the processor specific sections for availability. 


18.9.4 Reduced Skid PEBS 


Processors based on Goldmont Plus microarchitecture support the Reduced Skid PEBS feature described in Section 
18.5.3.1.2 on the IA32_PMCO counter. Although Extended PEBS adds support for generating PEBS records for 
precise events on additional general-purpose and fixed-function performance counters, those counters do not 
support the Reduced Skid PEBS feature. 


18.9.5 €PT-Friendly PEBS 


The 3rd generation Intel Xeon Scalable Family of processors based on Ice Lake microarchitecture (and later proces- 
sors) support VMX guest use of PEBS when the DS Area (including the PEBS Buffer and DS Management Area) is 
allocated from a paged pool of EPT pages. In such a configuration PEBS DS Area accesses may result in VM exits 
(e.g., EPT violations due to “lazy” EPT page-table entry propagation), and in such cases the PEBS record will not be 
lost but instead will “skid” to after the subsequent VM Entry back to the guest. For precise events the guest will 
observe that the record skid by one event occurrence, while for non-precise events the record will skid by one 
instruction. 
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CHAPTER 19 
8086 EMULATION 


IA-32 processors (beginning with the Intel386 processor) provide two ways to execute new or legacy programs 
that are assembled and/or compiled to run on an Intel 8086 processor: 


° Real-address mode. 
¢ Virtual-8086 mode. 


Figure 2-3 shows the relationship of these operating modes to protected mode and system management mode 
(SMM). 


When the processor is powered up or reset, it is placed in the real-address mode. This operating mode almost 
exactly duplicates the execution environment of the Intel 8086 processor, with some extensions. Virtually any 
program assembled and/or compiled to run on an Intel 8086 processor will run on an IA-32 processor in this mode. 


When running in protected mode, the processor can be switched to virtual-8086 mode to run 8086 programs. This 
mode also duplicates the execution environment of the Intel 8086 processor, with extensions. In virtual-8086 
mode, an 8086 program runs as a separate protected-mode task. Legacy 8086 programs are thus able to run 
under an operating system (such as Microsoft Windows*) that takes advantage of protected mode and to use 
protected-mode facilities, such as the protected-mode interrupt- and exception-handling facilities. Protected-mode 
multitasking permits multiple virtual-8086 mode tasks (with each task running a separate 8086 program) to be run 
on the processor along with other non-virtual-8086 mode tasks. 


This section describes both the basic real-address mode execution environment and the virtual-8086-mode execu- 
tion environment, available on the IA-32 processors beginning with the Intel386 processor. 


19.1 REAL-ADDRESS MODE 


The IA-32 architecture’s real-address mode runs programs written for the Intel 8086, Intel 8088, Intel 80186, and 
Intel 80188 processors, or for the real-address mode of the Intel 286, Intel386, Intel486, Pentium, P6 family, 
Pentium 4, and Intel Xeon processors. 


The execution environment of the processor in real-address mode is designed to duplicate the execution environ- 
ment of the Intel 8086 processor. To an 8086 program, a processor operating in real-address mode behaves like a 
high-speed 8086 processor. The principal features of this architecture are defined in Chapter 3, “Basic Execution 
Environment”, of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


The following is a summary of the core features of the real-address mode execution environment as would be seen 
by a program written for the 8086: 


¢ The processor supports a nominal 1-MByte physical address space (see Section 19.1.1, “Address Translation in 
Real-Address Mode”, for specific details). This address space is divided into segments, each of which can be up 
to 64 KBytes in length. The base of a segment is specified with a 16-bit segment selector, which is shifted left 
by 4 bits to form a 20-bit offset from address 0 in the address space. An operand within a segment is addressed 
with a 16-bit offset from the base of the segment. A physical address is thus formed by adding the offset to the 
20-bit segment base (see Section 19.1.1, “Address Translation in Real-Address Mode”). 


¢ All operands in “native 8086 code” are 8-bit or 16-bit values. (Operand size override prefixes can be used to 
access 32-bit operands.) 

e Eight 16-bit general-purpose registers are provided: AX, BX, CX, DX, SP, BP, SI, and DI. The extended 32 bit 
registers (EAX, EBX, ECX, EDX, ESP, EBP, ESI, and EDI) are accessible to programs that explicitly perform a size 
override operation. 

¢ Four segment registers are provided: CS, DS, SS, and ES. (The FS and GS registers are accessible to programs 
that explicitly access them.) The CS register contains the segment selector for the code segment; the DS and 
ES registers contain segment selectors for data segments; and the SS register contains the segment selector 
for the stack segment. 


° The 8086 16-bit instruction pointer (IP) is mapped to the lower 16-bits of the EIP register. Note this register is 
a 32-bit register and unintentional address wrapping may occur. 
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° The 16-bit FLAGS register contains status and control flags. (This register is mapped to the 16 least significant 
bits of the 32-bit EFLAGS register.) 


e All of the Intel 8086 instructions are supported (see Section 19.1.3, “Instructions Supported in Real-Address 
Mode”). 


e Asingle, 16-bit-wide stack is provided for handling procedure calls and invocations of interrupt and exception 
handlers. This stack is contained in the stack segment identified with the SS register. The SP (stack pointer) 
register contains an offset into the stack segment. The stack grows down (toward lower segment offsets) from 
the stack pointer. The BP (base pointer) register also contains an offset into the stack segment that can be used 
as a pointer to a parameter list. When a CALL instruction is executed, the processor pushes the current 
instruction pointer (the 16 least-significant bits of the EIP register and, on far calls, the current value of the CS 
register) onto the stack. On a return, initiated with a RET instruction, the processor pops the saved instruction 
pointer from the stack into the EIP register (and CS register on far returns). When an implicit call to an interrupt 
or exception handler is executed, the processor pushes the EIP, CS, and EFLAGS (low-order 16-bits only) 
registers onto the stack. On a return from an interrupt or exception handler, initiated with an IRET instruction, 
the processor pops the saved instruction pointer and EFLAGS image from the stack into the EIP, CS, and 
EFLAGS registers. 


¢ Asingle interrupt table, called the “interrupt vector table” or “interrupt table,” is provided for handling 
interrupts and exceptions (see Figure 19-2). The interrupt table (which has 4-byte entries) takes the place of 
the interrupt descriptor table (IDT, with 8-byte entries) used when handling protected-mode interrupts and 
exceptions. Interrupt and exception vector numbers provide an index to entries in the interrupt table. Each 
entry provides a pointer (called a “vector”) to an interrupt- or exception-handling procedure. See Section 
19.1.4, “Interrupt and Exception Handling”, for more details. It is possible for software to relocate the IDT by 
means of the LIDT instruction on IA-32 processors beginning with the Intel386 processor. 


° The x87 FPU is active and available to execute x87 FPU instructions in real-address mode. Programs written to 
run on the Intel 8087 and Intel 287 math coprocessors can be run in real-address mode without modification. 


The following extensions to the Intel 8086 execution environment are available in the IA-32 architecture's real- 
address mode. If backwards compatibility to Intel 286 and Intel 8086 processors is required, these features should 
not be used in new programs written to run in real-address mode. 


* Two additional segment registers (FS and GS) are available. 


° Many of the integer and system instructions that have been added to later IA-32 processors can be executed in 
real-address mode (see Section 19.1.3, “Instructions Supported in Real-Address Mode”). 


¢ The 32-bit operand prefix can be used in real-address mode programs to execute the 32-bit forms of instruc- 
tions. This prefix also allows real-address mode programs to use the processor’s 32-bit general-purpose 
registers. 


¢ The 32-bit address prefix can be used in real-address mode programs, allowing 32-bit offsets. 


The following sections describe address formation, registers, available instructions, and interrupt and exception 
handling in real-address mode. For information on I/O in real-address mode, see Chapter 19, “Input/Output”, of 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1. 


19.1.1 Address Translation in Real-Address Mode 


In real-address mode, the processor does not interpret segment selectors as indexes into a descriptor table; 
instead, it uses them directly to form linear addresses as the 8086 processor does. It shifts the segment selector 
left by 4 bits to form a 20-bit base address (see Figure 19-1). The offset into a segment is added to the base 
address to create a linear address that maps directly to the physical address space. 


When using 8086-style address translation, it is possible to specify addresses larger than 1 MByte. For example, 
with a segment selector value of FFFFH and an offset of FFFFH, the linear (and physical) address would be 1OFFEFH 
(1 megabyte plus 64 KBytes). The 8086 processor, which can form addresses only up to 20 bits long, truncates the 
high-order bit, thereby “wrapping” this address to FFEFH. When operating in real-address mode, however, the 
processor does not truncate such an address and uses it as a physical address. (Note, however, that for IA-32 
processors beginning with the Intel486 processor, the A20M# signal can be used in real-address mode to mask 
address line A20, thereby mimicking the 20-bit wrap-around behavior of the 8086 processor.) Care should be take 
to ensure that A20M# based address wrapping is handled correctly in multiprocessor based system. 
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19 4 3 0 
Base 16-bit Segment Selector 0000 
+ 19 16 15 0 
Offset | 0000 | 16-bit Effective Address 
~ 49 0 
Ponies 20-bit Linear Address 


Figure 19-1. Real-Address Mode Address Translation 


The IA-32 processors beginning with the Intel386 processor can generate 32-bit offsets using an address override 
prefix; however, in real-address mode, the value of a 32-bit offset may not exceed FFFFH without causing an 
exception. 


For full compatibility with Intel 286 real-address mode, pseudo-protection faults (interrupt 12 or 13) occur if a 32- 
bit offset is generated outside the range 0 through FFFFH. 


19.1.2 Registers Supported in Real-Address Mode 


The register set available in real-address mode includes all the registers defined for the 8086 processor plus the 
new registers introduced in later IA-32 processors, such as the FS and GS segment registers, the debug registers, 
the control registers, and the floating-point unit registers. The 32-bit operand prefix allows a real-address mode 
program to use the 32-bit general-purpose registers (EAX, EBX, ECX, EDX, ESP, EBP, ESI, and EDI). 


19.1.3. Instructions Supported in Real-Address Mode 


The following instructions make up the core instruction set for the 8086 processor. If backwards compatibility to 
the Intel 286 and Intel 8086 processors is required, only these instructions should be used in a new program 
written to run in real-address mode. 


° Move (MOV) instructions that move operands between general-purpose registers, segment registers, and 
between memory and general-purpose registers. 


¢ The exchange (XCHG) instruction. 

¢ Load segment register instructions LDS and LES. 

e ~=Arithmetic instructions ADD, ADC, SUB, SBB, MUL, IMUL, DIV, IDIV, INC, DEC, CMP, and NEG. 
° Logical instructions AND, OR, XOR, and NOT. 

¢ Decimal instructions DAA, DAS, AAA, AAS, AAM, and AAD. 

¢ Stack instructions PUSH and POP (to general-purpose registers and segment registers). 
° Type conversion instructions CWD, CDQ, CBW, and CWDE. 

¢ ~=Shift and rotate instructions SAL, SHL, SHR, SAR, ROL, ROR, RCL, and RCR. 

¢ TEST instruction. 

* Control instructions JMP, Jcc, CALL, RET, LOOP, LOOPE, and LOOPNE. 

¢ Interrupt instructions INT n, INTO, and IRET. 

¢ EFLAGS control instructions STC, CLC, CMC, CLD, STD, LAHF, SAHF, PUSHF, and POPF. 
¢ I/O instructions IN, INS, OUT, and OUTS. 

° Load effective address (LEA) instruction, and translate (XLATB) instruction. 
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¢ LOCK prefix. 

°® Repeat prefixes REP, REPE, REPZ, REPNE, and REPNZ. 
¢ Processor halt (HLT) instruction. 

¢ No operation (NOP) instruction. 


The following instructions, added to later IA-32 processors (some in the Intel 286 processor and the remainder in 
the Intel386 processor), can be executed in real-address mode, if backwards compatibility to the Intel 8086 
processor is not required. 


® Move (MOV) instructions that operate on the control and debug registers. 

¢ Load segment register instructions LSS, LFS, and LGS. 

* Generalized multiply instructions and multiply immediate data. 

¢ ~=Shift and rotate by immediate counts. 

® Stack instructions PUSHA, PUSHAD, POPA and POPAD, and PUSH immediate data. 
® Move with sign extension instructions MOVSX and MOVZX. 

¢ Long-displacement Jcc instructions. 

° Exchange instructions CMPXCHG, CMPXCHG8B, and XADD. 

¢ String instructions MOVS, CMPS, SCAS, LODS, and STOS. 


° Bit test and bit scan instructions BT, BTS, BTR, BTC, BSF, and BSR; the byte-set-on condition instruction SETcc; 
and the byte swap (BSWAP) instruction. 


* Double shift instructions SHLD and SHRD. 

¢ EFLAGS control instructions PUSHF and POPF. 
¢ ENTER and LEAVE control instructions. 

¢ BOUND instruction. 

* CPU identification (CPUID) instruction. 


° System instructions CLTS, INVD, WINVD, INVLPG, LGDT, SGDT, LIDT, SIDT, LMSW, SMSW, RDMSR, WRMSR, 
RDTSC, and RDPMC. 


Execution of any of the other IA-32 architecture instructions (not given in the previous two lists) in real-address 
mode result in an invalid-opcode exception (#UD) being generated. 


19.1.4 Interrupt and Exception Handling 


When operating in real-address mode, software must provide interrupt and exception-handling facilities that are 
separate from those provided in protected mode. Even during the early stages of processor initialization when the 
processor is still in real-address mode, elementary real-address mode interrupt and exception-handling facilities 
must be provided to ensure reliable operation of the processor, or the initialization code must ensure that no inter- 
rupts or exceptions will occur. 


The IA-32 processors handle interrupts and exceptions in real-address mode similar to the way they handle them 
in protected mode. When a processor receives an interrupt or generates an exception, it uses the vector number of 
the interrupt or exception as an index into the interrupt table. (In protected mode, the interrupt table is called the 
interrupt descriptor table (IDT), but in real-address mode, the table is usually called the interrupt vector 
table, or simply the interrupt table.) The entry in the interrupt vector table provides a pointer to an interrupt- or 
exception-handler procedure. (The pointer consists of a segment selector for a code segment and a 16-bit offset 
into the segment.) The processor performs the following actions to make an implicit call to the selected handler: 


1. Pushes the current values of the CS and EIP registers onto the stack. (Only the 16 least-significant bits of the 
EIP register are pushed.) 


Pushes the low-order 16 bits of the EFLAGS register onto the stack. 
Clears the IF flag in the EFLAGS register to disable interrupts. 
Clears the TF, RF, and AC flags, in the EFLAGS register. 


19-4 Vol. 3B 


8086 EMULATION 


5. Transfers program control to the location specified in the interrupt vector table. 


An IRET instruction at the end of the handler procedure reverses these steps to return program control to the inter- 
rupted program. Exceptions do not return error codes in real-address mode. 


The interrupt vector table is an array of 4-byte entries (see Figure 19-2). Each entry consists of a far pointer to a 
handler procedure, made up of a segment selector and an offset. The processor scales the interrupt or exception 
vector by 4 to obtain an offset into the interrupt table. Following reset, the base of the interrupt vector table is 
located at physical address 0 and its limit is set to 3FFH. In the Intel 8086 processor, the base address and limit of 
the interrupt vector table cannot be changed. In the later IA-32 processors, the base address and limit of the inter- 
rupt vector table are contained in the IDTR register and can be changed using the LIDT instruction. 


(For backward compatibility to Intel 8086 processors, the default base address and limit of the interrupt vector 
table should not be changed. ) 


A Up to Entry 255 A 
Entry 3 
12 
Entry 2 
8 
Entry 1 
4 
Segment Selector 2 
Interrupt Vector 0*—>>| 
Offset 0=<— 
15 0 
* Interrupt vector number 0 selects entry 0 
(called “interrupt vector 0”) in the interrupt IDTR 
vector table. Interrupt vector 0 in turn 
points to the start of the interrupt handler 
for interrupt 0. 


Figure 19-2. Interrupt Vector Table in Real-Address Mode 


Table 19-1 shows the interrupt and exception vectors that can be generated in real-address mode and virtual-8086 
mode, and in the Intel 8086 processor. See Chapter 6, “Interrupt and Exception Handling”, for a description of the 
exception conditions. 


19.2 VIRTUAL-8086 MODE 


Virtual-8086 mode is actually a special type of a task that runs in protected mode. When the operating-system or 
executive switches to a virtual-8086-mode task, the processor emulates an Intel 8086 processor. The execution 
environment of the processor while in the 8086-emulation state is the same as is described in Section 19.1, “Real- 
Address Mode” for real-address mode, including the extensions. The major difference between the two modes is 
that in virtual-8086 mode the 8086 emulator uses some protected-mode services (such as the protected-mode 
interrupt and exception-handling and paging facilities). 
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As in real-address mode, any new or legacy program that has been assembled and/or compiled to run on an Intel 
8086 processor will run in a virtual-8086-mode task. And several 8086 programs can be run as virtual-8086-mode 
tasks concurrently with normal protected-mode tasks, using the processor’s multitasking facilities. 

Table 19-1. Real-Address Mode Exceptions and Interrupts 


oe Description Real-Address Mode | Virtual-8086 Mode Intel 8086 Processor 
O. 

0 Divide Error (#DE) Yes Yes Yes 

1 Debug Exception (#DB) Yes Yes No 

2 NMI Interrupt Yes Yes Yes 

3 Breakpoint (#BP) Yes Yes Yes 

4 Overflow (#OF) Yes Yes Yes 

5 BOUND Range Exceeded (#BR) Yes Yes Reserved 
6 Invalid Opcode (#UD) Yes Yes Reserved 
7 Device Not Available (#NM) Yes Yes Reserved 
8 Double Fault (#DF) Yes Yes Reserved 
9 (Intel reserved. Do not use.) Reserved Reserved Reserved 
10 Invalid TSS (#TS) Reserved Yes Reserved 
11 Segment Not Present (#NP) Reserved Yes Reserved 
12 Stack Fault (#SS) Yes Yes Reserved 
13 General Protection (#GP)* Yes Yes Reserved 
14 Page Fault (#PF) Reserved Yes Reserved 
15 (Intel reserved. Do not use.) Reserved Reserved Reserved 
16 Floating-Point Error (#MF) Yes Yes Reserved 
17 Alignment Check (#AC) Reserved Yes Reserved 
18 Machine Check (#MC) Yes Yes Reserved 
19-31 (Intel reserved. Do not use.) Reserved Reserved Reserved 
32-255 User Defined Interrupts Yes Yes Yes 
NOTE: 


* In the real-address mode, vector 13 is the segment overrun exception. In protected and virtual-8086 modes, this exception cov- 
ers all general-protection error conditions, including traps to the virtual-8086 monitor from virtual-8086 mode. 


19.2.1. Enabling Virtual-8086 Mode 


The processor runs in virtual-8086 mode when the VM (virtual machine) flag in the EFLAGS register is set. This flag 
can only be set when the processor switches to a new protected-mode task or resumes virtual-8086 mode via an 
IRET instruction. 


System software cannot change the state of the VM flag directly in the EFLAGS register (for example, by using the 
POPFD instruction). Instead it changes the flag in the image of the EFLAGS register stored in the TSS or on the 
stack following a call to an interrupt- or exception-handler procedure. For example, software sets the VM flag in the 
EFLAGS image in the TSS when first creating a virtual-8086 task. 


The processor tests the VM flag under three general conditions: 
¢ When loading segment registers, to determine whether to use 8086-style address translation. 


¢ When decoding instructions, to determine which instructions are not supported in virtual-8086 mode and which 
instructions are sensitive to IOPL. 
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¢ When checking privileged instructions, on page accesses, or when performing other permission checks. 
(Virtual-8086 mode always executes at CPL 3.) 


19.2.2 Structure of a Virtual-8086 Task 


A virtual-8086-mode task consists of the following items: 
¢ A 32-bit TSS for the task. 

¢ The 8086 program. 

° A virtual-8086 monitor. 

* 8086 operating-system services. 


The TSS of the new task must be a 32-bit TSS, not a 16-bit TSS, because the 16-bit TSS does not load the most- 
significant word of the EFLAGS register, which contains the VM flag. All TSS’s, stacks, data, and code used to handle 
exceptions when in virtual-8086 mode must also be 32-bit segments. 


The processor enters virtual-8086 mode to run the 8086 program and returns to protected mode to run the virtual- 
8086 monitor. 


The virtual-8086 monitor is a 32-bit protected-mode code module that runs at a CPL of 0. The monitor consists of 
initialization, interrupt- and exception-handling, and I/O emulation procedures that emulate a personal computer 
or other 8086-based platform. Typically, the monitor is either part of or closely associated with the protected-mode 
general-protection (#GP) exception handler, which also runs at a CPL of 0. As with any protected-mode code 
module, code-segment descriptors for the virtual-8086 monitor must exist in the GDT or in the task’s LDT. The 
virtual-8086 monitor also may need data-segment descriptors so it can examine the IDT or other parts of the 8086 
program in the first 1 MByte of the address space. The linear addresses above 10FFEFH are available for the 
monitor, the operating system, and other system software. 


The 8086 operating-system services consists of a kernel and/or operating-system procedures that the 8086 
program makes calls to. These services can be implemented in either of the following two ways: 


°  =They can be included in the 8086 program. This approach is desirable for either of the following reasons: 
— The 8086 program code modifies the 8086 operating-system services. 


— There is not sufficient development time to merge the 8086 operating-system services into main operating 
system or executive. 


¢ They can be implemented or emulated in the virtual-8086 monitor. This approach is desirable for any of the 
following reasons: 
— The 8086 operating-system procedures can be more easily coordinated among several virtual-8086 tasks. 


— Memory can be saved by not duplicating 8086 operating-system procedure code for several virtual-8086 
tasks. 


— The 8086 operating-system procedures can be easily emulated by calls to the main operating system or 
executive. 


The approach chosen for implementing the 8086 operating-system services may result in different virtual-8086- 
mode tasks using different 8086 operating-system services. 


19.2.3. Paging of Virtual-8086 Tasks 


Even though a program running in virtual-8086 mode can use only 20-bit linear addresses, the processor converts 
these addresses into 32-bit linear addresses before mapping them to the physical address space. If paging is being 
used, the 8086 address space for a program running in virtual-8086 mode can be paged and located in a set of 
pages in physical address space. If paging is used, it is transparent to the program running in virtual-8086 mode 
just as it is for any task running on the processor. 


Paging is not necessary for a single virtual-8086-mode task, but paging is useful or necessary in the following situ- 
ations: 
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When running multiple virtual-8086-mode tasks. Here, paging allows the lower 1 MByte of the linear address 
space for each virtual-8086-mode task to be mapped to a different physical address location. 


When emulating the 8086 address-wraparound that occurs at 1 MByte. When using 8086-style address trans- 
lation, it is possible to specify addresses larger than 1 MByte. These addresses automatically wraparound in the 
Intel 8086 processor (see Section 19.1.1, “Address Translation in Real-Address Mode”). If any 8086 programs 
depend on address wraparound, the same effect can be achieved in a virtual-8086-mode task by mapping the 
linear addresses between 100000H and 110000H and linear addresses between 0 and 10000H to the same 
physical addresses. 


When sharing the 8086 operating-system services or ROM code that is common to several 8086 programs 
running as different 8086-mode tasks. 


When redirecting or trapping references to memory-mapped I/O devices. 


19.2.4 Protection within a Virtual-8086 Task 


Protection is not enforced between the segments of an 8086 program. Either of the following techniques can be 
used to protect the system software running in a virtual-8086-mode task from the 8086 program: 


Reserve the first 1 MByte plus 64 KBytes of each task’s linear address space for the 8086 program. An 8086 
processor task cannot generate addresses outside this range. 


Use the U/S flag of page-table entries to protect the virtual-8086 monitor and other system software in the 
virtual-8086 mode task space. When the processor is in virtual-8086 mode, the CPL is 3. Therefore, an 8086 
processor program has only user privileges. If the pages of the virtual-8086 monitor have supervisor privilege, 
they cannot be accessed by the 8086 program. 


19.2.5 Entering Virtual-8086 Mode 


Figure 19-3 summarizes the methods of entering and leaving virtual-8086 mode. The processor switches to 
virtual-8086 mode in either of the following situations: 


Task switch when the VM flag is set to 1 in the EFLAGS register image stored in the TSS for the task. Here the 
task switch can be initiated in either of two ways: 


— ACALL or JMP instruction. 
— An IRET instruction, where the NT flag in the EFLAGS image is set to 1. 


Return from a protected-mode interrupt or exception handler when the VM flag is set to 1 in the EFLAGS 
register image on the stack. 


When a task switch is used to enter virtual-8086 mode, the TSS for the virtual-8086-mode task must be a 32-bit 
TSS. (If the new TSS is a 16-bit TSS, the upper word of the EFLAGS register is not in the TSS, causing the processor 
to clear the VM flag when it loads the EFLAGS register.) The processor updates the VM flag prior to loading the 
segment registers from their images in the new TSS. The new setting of the VM flag determines whether the 
processor interprets the contents of the segment registers as 8086-style segment selectors or protected-mode 
segment selectors. When the VM flag is set, the segment registers are loaded from the TSS, using 8086-style 
address translation to form base addresses. 


See Section 19.3, “Interrupt and Exception Handling in Virtual-8086 Mode”, for information on entering virtual- 
8086 mode on a return from an interrupt or exception handler. 


19-8 Vol. 3B 


8086 EMULATION 


. Real Mode 
Code 
Real-Address r 
Mode 7 PE=0 or 
PES! RESET 
Y Task Switch 
ask SWIC Protected- CALL 
eats Protected- | ~VM=0 Mode Interrupt |= |-Virtual-8086 
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Redirect Interrupt to 8086 Program 
Interrupt or Exception Handler 


NOTES: 

1. Task switch carried out in either of two ways: 

- CALL or JMP where the VM flag in the EFLAGS image is 1. 
- IRET where VM is 1 and NT is 1. 

2. Hardware interrupt or exception; software interrupt (INT n) when IOPL is 3. 

3. General-protection exception caused by software interrupt (INT n), IRET, 
POPF, PUSHF, IN, or OUT when IOPL is less than 3. 

4. Normal return from protected-mode interrupt or exception handler. 

5. Areturn from the 8086 monitor to redirect an interrupt or exception back 
to an interrupt or exception handler in the 8086 program running in virtual- 
8086 mode. 

6. Internal redirection of a software interrupt (INT n) when VME is 1, 

IOPL is <3, and the redirection bit is 1. 


Figure 19-3. Entering and Leaving Virtual-8086 Mode 


19.2.6 Leaving Virtual-8086 Mode 


The processor can leave the virtual-8086 mode only through an interrupt or exception. The following are situations 
where an interrupt or exception will lead to the processor leaving virtual-8086 mode (see Figure 19-3): 


¢ The processor services a hardware interrupt generated to signal the suspension of execution of the virtual- 
8086 application. This hardware interrupt may be generated by a timer or other external mechanism. Upon 
receiving the hardware interrupt, the processor enters protected mode and switches to a protected-mode (or 
another virtual-8086 mode) task either through a task gate in the protected-mode IDT or through a trap or 
interrupt gate that points to a handler that initiates a task switch. A task switch from a virtual-8086 task to 
another task loads the EFLAGS register from the TSS of the new task. The value of the VM flag in the new 
EFLAGS determines if the new task executes in virtual-8086 mode or not. 


¢ The processor services an exception caused by code executing the virtual-8086 task or services a hardware 
interrupt that “belongs to” the virtual-8086 task. Here, the processor enters protected mode and services the 
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exception or hardware interrupt through the protected-mode IDT (normally through an interrupt or trap gate) 
and the protected-mode exception- and interrupt-handlers. The processor may handle the exception or 
interrupt within the context of the virtual 8086 task and return to virtual-8086 mode on a return from the 
handler procedure. The processor may also execute a task switch and handle the exception or interrupt in the 
context of another task. 


¢ The processor services a software interrupt generated by code executing in the virtual-8086 task (such as a 
software interrupt to call a MS-DOS* operating system routine). The processor provides several methods of 
handling these software interrupts, which are discussed in detail in Section 19.3.3, “Class 3—Software 
Interrupt Handling in Virtual-8086 Mode”. Most of them involve the processor entering protected mode, often 
by means of a general-protection (#GP) exception. In protected mode, the processor can send the interrupt to 
the virtual-8086 monitor for handling and/or redirect the interrupt back to the application program running in 
virtual-8086 mode task for handling. 


IA-32 processors that incorporate the virtual mode extension (enabled with the VME flag in control register 
CR4) are capable of redirecting software-generated interrupts back to the program’s interrupt handlers without 
leaving virtual-8086 mode. See Section 19.3.3.4, “Method 5: Software Interrupt Handling”, for more 
information on this mechanism. 


¢ A hardware reset initiated by asserting the RESET or INIT pin is a special kind of interrupt. When a RESET or 
INIT is signaled while the processor is in virtual-8086 mode, the processor leaves virtual-8086 mode and enters 
real-address mode. 


¢ Execution of the HLT instruction in virtual-8086 mode will cause a general-protection (GP#) fault, which the 
protected-mode handler generally sends to the virtual-8086 monitor. The virtual-8086 monitor then 
determines the correct execution sequence after verifying that it was entered as a result of a HLT execution. 


See Section 19.3, “Interrupt and Exception Handling in Virtual-8086 Mode”, for information on leaving virtual-8086 
mode to handle an interrupt or exception generated in virtual-8086 mode. 


19.2.7 Sensitive Instructions 


When an IA-32 processor is running in virtual-8086 mode, the CLI, STI, PUSHF, POPF, INT n, and IRET instructions 
are sensitive to IOPL. The IN, INS, OUT, and OUTS instructions, which are sensitive to IOPL in protected mode, are 
not sensitive in virtual-8086 mode. 


The CPL is always 3 while running in virtual-8086 mode; if the IOPL is less than 3, an attempt to use the IOPL-sensi- 
tive instructions listed above triggers a general-protection exception (#GP). These instructions are sensitive to 
IOPL to give the virtual-8086 monitor a chance to emulate the facilities they affect. 


19.2.8 Virtual-8086 Mode I/O 


Many 8086 programs written for non-multitasking systems directly access I/O ports. This practice may cause prob- 
lems in a multitasking environment. If more than one program accesses the same port, they may interfere with 
each other. Most multitasking systems require application programs to access I/O ports through the operating 
system. This results in simplified, centralized control. 


The processor provides I/O protection for creating I/O that is compatible with the environment and transparent to 
8086 programs. Designers may take any of several possible approaches to protecting I/O ports: 


¢ Protect the I/O address space and generate exceptions for all attempts to perform I/O directly. 
¢ Let the 8086 program perform I/O directly. 

°® Generate exceptions on attempts to access specific I/O ports. 

* Generate exceptions on attempts to access specific memory-mapped I/O ports. 


The method of controlling access to I/O ports depends upon whether they are I/O-port mapped or memory 
mapped. 
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19.2.8.1  1/0-Port-Mapped I/O 


The I/O permission bit map in the TSS can be used to generate exceptions on attempts to access specific I/O port 
addresses. The I/O permission bit map of each virtual-8086-mode task determines which I/O addresses generate 
exceptions for that task. Because each task may have a different I/O permission bit map, the addresses that 
generate exceptions for one task may be different from the addresses for another task. This differs from protected 
mode in which, if the CPL is less than or equal to the IOPL, I/O access is allowed without checking the I/O permis- 
sion bit map. See Chapter 19, “Input/Output”, in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 1, for more information about the I/O permission bit map. 


19.2.8.2 Memory-Mapped I/O 


In systems which use memory-mapped I/O, the paging facilities of the processor can be used to generate excep- 
tions for attempts to access I/O ports. The virtual-8086 monitor may use paging to control memory-mapped I/O in 
these ways: 


° Map part of the linear address space of each task that needs to perform I/O to the physical address space 
where I/O ports are placed. By putting the I/O ports at different addresses (in different pages), the paging 
mechanism can enforce isolation between tasks. 


° Map part of the linear address space to pages that are not-present. This generates an exception whenever a 
task attempts to perform I/O to those pages. System software then can interpret the I/O operation being 
attempted. 


Software emulation of the I/O space may require too much operating system intervention under some conditions. 
In these cases, it may be possible to generate an exception for only the first attempt to access I/O. The system 
software then may determine whether a program can be given exclusive control of I/O temporarily, the protection 
of the I/O space may be lifted, and the program allowed to run at full speed. 


19.2.8.3. Special I/O Buffers 


Buffers of intelligent controllers (for example, a bit-mapped frame buffer) also can be emulated using page 
mapping. The linear space for the buffer can be mapped to a different physical space for each virtual-8086-mode 
task. The virtual-8086 monitor then can control which virtual buffer to copy onto the real buffer in the physical 
address space. 


19.3. INTERRUPT AND EXCEPTION HANDLING IN VIRTUAL-8086 MODE 


When the processor receives an interrupt or detects an exception condition while in virtual-8086 mode, it invokes 
an interrupt or exception handler, just as it does in protected or real-address mode. The interrupt or exception 
handler that is invoked and the mechanism used to invoke it depends on the class of interrupt or exception that has 
been detected or generated and the state of various system flags and fields. 


In virtual-8086 mode, the interrupts and exceptions are divided into three classes for the purposes of handling: 


° Class 1 — All processor-generated exceptions and all hardware interrupts, including the NMI interrupt and the 
hardware interrupts sent to the processor’s external interrupt delivery pins. All class 1 exceptions and 
interrupts are handled by the protected-mode exception and interrupt handlers. 


° Class 2 — Special case for maskable hardware interrupts (Section 6.3.2, ‘“Maskable Hardware Interrupts”) 
when the virtual mode extensions are enabled. 


* Class 3 — All software-generated interrupts, that is interrupts generated with the INT n instruction?. 


The method the processor uses to handle class 2 and 3 interrupts depends on the setting of the following flags and 
fields: 


¢ IOPL field (bits 12 and 13 in the EFLAGS register) — Controls how class 3 software interrupts are handled 
when the processor is in virtual-8086 mode (see Section 2.3, “System Flags and Fields in the EFLAGS 


1. The INT 3 instruction is a special case (See the description of the INT ninstruction in Chapter 3, “Instruction Set Reference, A-L", of 
the Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 2A). 
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Register”). This field also controls the enabling of the VIF and VIP flags in the EFLAGS register when the VME 
flag is set. The VIF and VIP flags are provided to assist in the handling of class 2 maskable hardware interrupts. 


© VME flag (bit O in control register CR4) — Enables the virtual mode extension for the processor when set 
(see Section 2.5, “Control Registers”). 


* Software interrupt redirection bit map (32 bytes in the TSS, see Figure 19-5) — Contains 256 flags 
that indicates how class 3 software interrupts should be handled when they occur in virtual-8086 mode. A 
software interrupt can be directed either to the interrupt and exception handlers in the currently running 8086 
program or to the protected-mode interrupt and exception handlers. 


¢ The virtual interrupt flag (VIF) and virtual interrupt pending flag (VIP) in the EFLAGS register — 
Provides virtual interrupt support for the handling of class 2 maskable hardware interrupts (see Section 
19.3.2, “Class 2—Maskable Hardware Interrupt Handling in Virtual-8086 Mode Using the Virtual Interrupt 
Mechanism”). 


NOTE 


The VME flag, software interrupt redirection bit map, and VIF and VIP flags are only available in 
IA-32 processors that support the virtual mode extensions. These extensions were introduced in 
the IA-32 architecture with the Pentium processor. 


The following sections describe the actions that processor takes and the possible actions of interrupt and exception 
handlers for the two classes of interrupts described in the previous paragraphs. These sections describe three 
possible types of interrupt and exception handlers: 


¢ Protected-mode interrupt and exceptions handlers — These are the standard handlers that the processor 
calls through the protected-mode IDT. 


* Virtual-8086 monitor interrupt and exception handlers — These handlers are resident in the virtual-8086 
monitor, and they are commonly accessed through a general-protection exception (#GP, interrupt 13) that is 
directed to the protected-mode general-protection exception handler. 


* 8086 program interrupt and exception handlers — These handlers are part of the 8086 program that is 
running in virtual-8086 mode. 


The following sections describe how these handlers are used, depending on the selected class and method of inter- 
rupt and exception handling. 


19.3.1 Class 1—Hardware Interrupt and Exception Handling in Virtual-8086 Mode 


In virtual-8086 mode, the Pentium, P6 family, Pentium 4, and Intel Xeon processors handle hardware interrupts 
and exceptions in the same manner as they are handled by the Intel486 and Intel386 processors. They invoke the 
protected-mode interrupt or exception handler that the interrupt or exception vector points to in the IDT. Here, the 
IDT entry must contain either a 32-bit trap or interrupt gate or a task gate. The following sections describe various 
ways that a virtual-8086 mode interrupt or exception can be handled after the protected-mode handler has been 
invoked. 


See Section 19.3.2, “Class 2—Maskable Hardware Interrupt Handling in Virtual-8086 Mode Using the Virtual Inter- 
rupt Mechanism”, for a description of the virtual interrupt mechanism that is available for handling maskable hard- 
ware interrupts while in virtual-8086 mode. When this mechanism is either not available or not enabled, maskable 
hardware interrupts are handled in the same manner as exceptions, as described in the following sections. 


19.3.1.1 Handling an Interrupt or Exception Through a Protected-Mode Trap or Interrupt Gate 


When an interrupt or exception vector points to a 32-bit trap or interrupt gate in the IDT, the gate must in turn point 
to a nonconforming, privilege-level 0, code segment. When accessing this code segment, processor performs the 
following steps. 


1. Switches to 32-bit protected mode and privilege level 0. 


2. Saves the state of the processor on the privilege-level 0 stack. The states of the EIP, CS, EFLAGS, ESP, SS, ES, 
DS, FS, and GS registers are saved (see Figure 19-4). 
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Clears the segment registers. Saving the DS, ES, FS, and GS registers on the stack and then clearing the 
registers lets the interrupt or exception handler safely save and restore these registers regardless of the type 
segment selectors they contain (protected-mode or 8086-style). The interrupt and exception handlers, which 
may be called in the context of either a protected-mode task or a virtual-8086-mode task, can use the same 
code sequences for saving and restoring the registers for any task. Clearing these registers before execution of 
the IRET instruction does not cause a trap in the interrupt handler. Interrupt procedures that expect values in 
the segment registers or that return values in the segment registers must use the register images saved on the 
stack for privilege level 0. 


Clears VM, NT, RF and TF flags (in the EFLAGS register). If the gate is an interrupt gate, clears the IF flag. 
Begins executing the selected interrupt or exception handler. 


If the trap or interrupt gate references a procedure in a conforming segment or in a segment at a privilege level 
other than O, the processor generates a general-protection exception (#GP). Here, the error code is the segment 
selector of the code segment to which a call was attempted. 


Without Error Code With Error Code 
Unused <— a from Unused << =a from 
Old GS Old GS 
Old FS Old FS 
Old DS Old DS 
Old ES Old ES 
Old SS Old SS 
Old ESP Old ESP 
Old EFLAGS Old EFLAGS 
Old CS Old CS 
Old EIP ~<— New ESP Old EIP 
Error Code <— New ESP 


Figure 19-4. Privilege Level O Stack After Interrupt or 
Exception in Virtual-8086 Mode 


Interrupt and exception handlers can examine the VM flag on the stack to determine if the interrupted procedure 
was running in virtual-8086 mode. If so, the interrupt or exception can be handled in one of three ways: 


The protected-mode interrupt or exception handler that was called can handle the interrupt or exception. 


The protected-mode interrupt or exception handler can call the virtual-8086 monitor to handle the interrupt or 
exception. 


The virtual-8086 monitor (if called) can in turn pass control back to the 8086 program’s interrupt and exception 
handler. 


If the interrupt or exception is handled with a protected-mode handler, the handler can return to the interrupted 
program in virtual-8086 mode by executing an IRET instruction. This instruction loads the EFLAGS and segment 
registers from the images saved in the privilege level 0 stack (see Figure 19-4). A set VM flag in the EFLAGS image 
causes the processor to switch back to virtual-8086 mode. The CPL at the time the IRET instruction is executed 
must be 0, otherwise the processor does not change the state of the VM flag. 
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The virtual-8086 monitor runs at privilege level 0, like the protected-mode interrupt and exception handlers. It is 
commonly closely tied to the protected-mode general-protection exception (#GP, vector 13) handler. If the 
protected-mode interrupt or exception handler calls the virtual-8086 monitor to handle the interrupt or exception, 
the return from the virtual-8086 monitor to the interrupted virtual-8086 mode program requires two return 
instructions: a RET instruction to return to the protected-mode handler and an IRET instruction to return to the 
interrupted program. 


The virtual-8086 monitor has the option of directing the interrupt and exception back to an interrupt or exception 
handler that is part of the interrupted 8086 program, as described in Section 19.3.1.2, “Handling an Interrupt or 
Exception With an 8086 Program Interrupt or Exception Handler”. 


19.3.1.2 Handling an Interrupt or Exception With an 8086 Program Interrupt or Exception Handler 


Because it was designed to run on an 8086 processor, an 8086 program running in a virtual-8086-mode task 
contains an 8086-style interrupt vector table, which starts at linear address 0. If the virtual-8086 monitor correctly 
directs an interrupt or exception vector back to the virtual-8086-mode task it came from, the handlers in the 8086 
program can handle the interrupt or exception. The virtual-8086 monitor must carry out the following steps to send 
an interrupt or exception back to the 8086 program: 


1. Use the 8086 interrupt vector to locate the appropriate handler procedure in the 8086 program interrupt table. 


2. Store the EFLAGS (low-order 16 bits only), CS and EIP values of the 8086 program on the privilege-level 3 
stack. This is the stack that the virtual-8086-mode task is using. (The 8086 handler may use or modify this 
information. ) 


3. Change the return link on the privilege-level 0 stack to point to the privilege-level 3 handler procedure. 
Execute an IRET instruction to pass control to the 8086 program handler. 


5. When the IRET instruction from the privilege-level 3 handler triggers a general-protection exception (#GP) and 
thus effectively again calls the virtual-8086 monitor, restore the return link on the privilege-level 0 stack to 
point to the original, interrupted, privilege-level 3 procedure. 


6. Copy the low order 16 bits of the EFLAGS image from the privilege-level 3 stack to the privilege-level 0 stack 
(because some 8086 handlers modify these flags to return information to the code that caused the interrupt). 


7. Execute an IRET instruction to pass control back to the interrupted 8086 program. 


Note that if an operating system intends to support all 8086 MS-DOS-based programs, it is necessary to use the 
actual 8086 interrupt and exception handlers supplied with the program. The reason for this is that some programs 
modify their own interrupt vector table to substitute (or hook in series) their own specialized interrupt and excep- 
tion handlers. 


19.3.1.3. Handling an Interrupt or Exception Through a Task Gate 


When an interrupt or exception vector points to a task gate in the IDT, the processor performs a task switch to the 
selected interrupt- or exception-handling task. The following actions are carried out as part of this task switch: 


1. The EFLAGS register with the VM flag set is saved in the current TSS. 


2. The link field in the TSS of the called task is loaded with the segment selector of the TSS for the interrupted 
virtual-8086-mode task. 


3. The EFLAGS register is loaded from the image in the new TSS, which clears the VM flag and causes the 
processor to switch to protected mode. 


4. The NT flag in the EFLAGS register is set. 
5. The processor begins executing the selected interrupt- or exception-handler task. 


When an IRET instruction is executed in the handler task and the NT flag in the EFLAGS register is set, the proces- 
sors switches from a protected-mode interrupt- or exception-handler task back to a virtual-8086-mode task. Here, 
the EFLAGS and segment registers are loaded from images saved in the TSS for the virtual-8086-mode task. If the 
VM flag is set in the EFLAGS image, the processor switches back to virtual-8086 mode on the task switch. The CPL 
at the time the IRET instruction is executed must be 0, otherwise the processor does not change the state of the 
VM flag. 
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19.3.2 Class 2—Maskable Hardware Interrupt Handling in Virtual-8086 Mode Using the 
Virtual Interrupt Mechanism 


Maskable hardware interrupts are those interrupts that are delivered through the INTR# pin or through an inter- 
rupt request to the local APIC (see Section 6.3.2, “Maskable Hardware Interrupts”). These interrupts can be inhib- 
ited (masked) from interrupting an executing program or task by clearing the IF flag in the EFLAGS register. 


When the VME flag in control register CR4 is set and the IOPL field in the EFLAGS register is less than 3, two addi- 
tional flags are activated in the EFLAGS register: 


¢  =VIF (virtual interrupt) flag, bit 19 of the EFLAGS register. 
¢ VIP (virtual interrupt pending) flag, bit 20 of the EFLAGS register. 


These flags provide the virtual-8086 monitor with more efficient control over handling maskable hardware inter- 
rupts that occur during virtual-8086 mode tasks. They also reduce interrupt-handling overhead, by eliminating the 
need for all IF related operations (such as PUSHF, POPF, CLI, and STI instructions) to trap to the virtual-8086 
monitor. The purpose and use of these flags are as follows. 


NOTE 


The VIF and VIP flags are only available in IA-32 processors that support the virtual mode 
extensions. These extensions were introduced in the IA-32 architecture with the Pentium 
processor. When this mechanism is either not available or not enabled, maskable hardware 
interrupts are handled as class 1 interrupts. Here, if VIF and VIP flags are needed, the virtual-8086 
monitor can implement them in software. 


Existing 8086 programs commonly set and clear the IF flag in the EFLAGS register to enable and disable maskable 
hardware interrupts, respectively; for example, to disable interrupts while handling another interrupt or an excep- 
tion. This practice works well in single task environments, but can cause problems in multitasking and multiple- 
processor environments, where it is often desirable to prevent an application program from having direct control 
over the handling of hardware interrupts. When using earlier IA-32 processors, this problem was often solved by 
creating a virtual IF flag in software. The IA-32 processors (beginning with the Pentium processor) provide hard- 
ware support for this virtual IF flag through the VIF and VIP flags. 


The VIF flag is a virtualized version of the IF flag, which an application program running from within a virtual-8086 
task can used to control the handling of maskable hardware interrupts. When the VIF flag is enabled, the CLI and 
STI instructions operate on the VIF flag instead of the IF flag. When an 8086 program executes the CLI instruction, 
the processor clears the VIF flag to request that the virtual-8086 monitor inhibit maskable hardware interrupts 
from interrupting program execution; when it executes the STI instruction, the processor sets the VIF flag 
requesting that the virtual-8086 monitor enable maskable hardware interrupts for the 8086 program. But actually 
the IF flag, managed by the operating system, always controls whether maskable hardware interrupts are enabled. 
Also, if under these circumstances an 8086 program tries to read or change the IF flag using the PUSHF or POPF 
instructions, the processor will change the VIF flag instead, leaving IF unchanged. 


The VIP flag provides software a means of recording the existence of a deferred (or pending) maskable hardware 
interrupt. This flag is read by the processor but never explicitly written by the processor; it can only be written by 
software. 


If the IF flag is set and the VIF and VIP flags are enabled, and the processor receives a maskable hardware inter- 
rupt (interrupt vector 0 through 255), the processor performs and the interrupt handler software should perform 
the following operations: 


1. The processor invokes the protected-mode interrupt handler for the interrupt received, as described in the 
following steps. These steps are almost identical to those described for method 1 interrupt and exception 
handling in Section 19.3.1.1, “Handling an Interrupt or Exception Through a Protected-Mode Trap or Interrupt 
Gate”: 


a. Switches to 32-bit protected mode and privilege level 0. 


b. Saves the state of the processor on the privilege-level 0 stack. The states of the EIP, CS, EFLAGS, ESP, SS, 
ES, DS, FS, and GS registers are saved (see Figure 19-4). 


c. Clears the segment registers. 
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d. Clears the VM flag in the EFLAGS register. 
e. Begins executing the selected protected-mode interrupt handler. 


2. The recommended action of the protected-mode interrupt handler is to read the VM flag from the EFLAGS 
image on the stack. If this flag is set, the handler makes a call to the virtual-8086 monitor. 


3. The virtual-8086 monitor should read the VIF flag in the EFLAGS register. 


— Ifthe VIF flag is clear, the virtual-8086 monitor sets the VIP flag in the EFLAGS image on the stack to 
indicate that there is a deferred interrupt pending and returns to the protected-mode handler. 


— Ifthe VIF flag is set, the virtual-8086 monitor can handle the interrupt if it “belongs” to the 8086 program 
running in the interrupted virtual-8086 task; otherwise, it can call the protected-mode interrupt handler to 
handle the interrupt. 


4. The protected-mode handler executes a return to the program executing in virtual-8086 mode. 
5. Upon returning to virtual-8086 mode, the processor continues execution of the 8086 program. 


When the 8086 program is ready to receive maskable hardware interrupts, it executes the STI instruction to set the 
VIF flag (enabling maskable hardware interrupts). Prior to setting the VIF flag, the processor automatically checks 
the VIP flag and does one of the following, depending on the state of the flag: 


° Ifthe VIP flag is clear (indicating no pending interrupts), the processor sets the VIF flag. 


e If the VIP flag is set (indicating a pending interrupt), the processor generates a general-protection exception 
(#GP). 


The recommended action of the protected-mode general-protection exception handler is to then call the virtual- 
8086 monitor and let it handle the pending interrupt. After handling the pending interrupt, the typical action of the 
virtual-8086 monitor is to clear the VIP flag and set the VIF flag in the EFLAGS image on the stack, and then 
execute a return to the virtual-8086 mode. The next time the processor receives a maskable hardware interrupt, it 
will then handle it as described in steps 1 through 5 earlier in this section. 


If the processor finds that both the VIF and VIP flags are set at the beginning of an instruction, it generates a 
general-protection exception. This action allows the virtual-8086 monitor to handle the pending interrupt for the 
virtual-8086 mode task for which the VIF flag is enabled. Note that this situation can only occur immediately 
following execution of a POPF or IRET instruction or upon entering a virtual-8086 mode task through a task switch. 


Note that the states of the VIF and VIP flags are not modified in real-address mode or during transitions between 
real-address and protected modes. 


NOTE 


The virtual interrupt mechanism described in this section is also available for use in protected 
mode, see Section 19.4, “Protected-Mode Virtual Interrupts”. 


19.3.3. Class 3—Software Interrupt Handling in Virtual-8086 Mode 


When the processor receives a software interrupt (an interrupt generated with the INT rn instruction) while in 
virtual-8086 mode, it can use any of six different methods to handle the interrupt. The method selected depends 
on the settings of the VME flag in control register CR4, the IOPL field in the EFLAGS register, and the software inter- 
rupt redirection bit map in the TSS. Table 19-2 lists the six methods of handling software interrupts in virtual-8086 
mode and the respective settings of the VME flag, IOPL field, and the bits in the interrupt redirection bit map for 
each method. The table also summarizes the various actions the processor takes for each method. 


The VME flag enables the virtual mode extensions for the Pentium and later IA-32 processors. When this flag is 
clear, the processor responds to interrupts and exceptions in virtual-8086 mode in the same manner as an Intel386 
or Intel486 processor does. When this flag is set, the virtual mode extension provides the following enhancements 
to virtual-8086 mode: 


° Speeds up the handling of software-generated interrupts in virtual-8086 mode by allowing the processor to 
bypass the virtual-8086 monitor and redirect software interrupts back to the interrupt handlers that are part of 
the currently running 8086 program. 


* Supports virtual interrupts for software written to run on the 8086 processor. 
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The IOPL value interacts with the VME flag and the bits in the interrupt redirection bit map to determine how 
specific software interrupts should be handled. 


The software interrupt redirection bit map (see Figure 19-5) is a 32-byte field in the TSS. This map is located 
directly below the I/O permission bit map in the TSS. Each bit in the interrupt redirection bit map is mapped to an 
interrupt vector. Bit 0 in the interrupt redirection bit map (which maps to vector zero in the interrupt table) is 
located at the I/O base map address in the TSS minus 32 bytes. When a bit in this bit map is set, it indicates that 
the associated software interrupt (interrupt generated with an INT rn instruction) should be handled through the 
protected-mode IDT and interrupt and exception handlers. When a bit in this bit map is clear, the processor redi- 
rects the associated software interrupt back to the interrupt table in the 8086 program (located at linear address 0 
in the program’s address space). 


NOTE 


The software interrupt redirection bit map does not affect hardware generated interrupts and 
exceptions. Hardware generated interrupts and exceptions are always handled by the protected- 
mode interrupt and exception handlers. 


Table 19-2. Software Interrupt Handling Methods While in Virtual-8086 Mode 
Bit in 
Redir. 
Method | VME | IOPL | Bitmap* Processor Action 


1 0 3 X Interrupt directed to a protected-mode interrupt handler: 


* Switches to privilege-level O stack 
» Pushes GS, FS, DS and ES onto privilege-level 0 stack 

= Pushes SS, ESP, EFLAGS, CS and EIP of interrupted task onto privilege-level O stack 
* Clears VM, RF, NT, and TF flags 


If serviced through interrupt gate, clears IF flag 
Clears GS, FS, DS and ES to 0 
Sets CS and EIP from interrupt gate 


2 0) <3 X Interrupt directed to protected-mode general-protection exception (#GP) handler. 


3 1 <3 1 Interrupt directed to a protected-mode general-protection exception (#GP) handler; VIF and VIP 
flag support for handling class 2 maskable hardware interrupts. 


4 1 3 1 Interrupt directed to protected-mode interrupt handler: (see method 1 processor action). 


5 1 3 0 Interrupt redirected to 8086 program interrupt handler: 


* Pushes EFLAGS 

* Pushes CS and EIP (lower 16 bits only) 

* Clears IF flag 

* Clears TF flag 

* Loads CS and EIP (lower 16 bits only) from selected entry in the interrupt vector table of the 
current virtual-8086 task 


6 1 <3 0 Interrupt redirected to 8086 program interrupt handler; VIF and VIP flag support for handling class 
2 maskable hardware interrupts: 


* Pushes EFLAGS with IOPL set to 3 and VIF copied to IF 

* Pushes CS and EIP (lower 16 bits only) 

* Clears the VIF flag 

* Clears TF flag 

* Loads CS and EIP (lower 16 bits only) from selected entry in the interrupt vector table of the 
current virtual-8086 task 


NOTE: 


* When set to O, software interrupt is redirected back to the 8086 program interrupt handler; when set to 1, interrupt is directed to 
protected-mode handler. 
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Figure 19-5. Software Interrupt Redirection Bit Map in TSS 


Redirecting software interrupts back to the 8086 program potentially speeds up interrupt handling because a 
switch back and forth between virtual-8086 mode and protected mode is not required. This latter interrupt- 
handling technique is particularly useful for 8086 operating systems (such as MS-DOS) that use the INT rn instruc- 
tion to call operating system procedures. 


The CPUID instruction can be used to verify that the virtual mode extension is implemented on the processor. Bit 1 
of the feature flags register (EDX) indicates the availability of the virtual mode extension (see “CPUID—CPU Iden- 
tification” in Chapter 3, “Instruction Set Reference, A-L”, of the Inte/® 64 and IA-32 Architectures Software Devel- 
oper’s Manual, Volume 2A). 


The following sections describe the six methods (or mechanisms) for handling software interrupts in virtual-8086 
mode. See Section 19.3.2, “Class 2—Maskable Hardware Interrupt Handling in Virtual-8086 Mode Using the Virtual 
Interrupt Mechanism”, for a description of the use of the VIF and VIP flags in the EFLAGS register for handling 
maskable hardware interrupts. 


19.3.3.1 Method 1: Software Interrupt Handling 


When the VME flag in control register CR4 is clear and the IOPL field is 3, a Pentium or later IA-32 processor 
handles software interrupts in the same manner as they are handled by an Intel386 or Intel486 processor. It 
executes an implicit call to the interrupt handler in the protected-mode IDT pointed to by the interrupt vector. See 
Section 19.3.1, “Class 1—Hardware Interrupt and Exception Handling in Virtual-8086 Mode”, for a complete 
description of this mechanism and its possible uses. 


19.3.3.2 Methods 2 and 3: Software Interrupt Handling 


When a software interrupt occurs in virtual-8086 mode and the method 2 or 3 conditions are present, the processor 
generates a general-protection exception (#GP). Method 2 is enabled when the VME flag is set to 0 and the IOPL 
value is less than 3. Here the IOPL value is used to bypass the protected-mode interrupt handlers and cause any 
software interrupt that occurs in virtual-8086 mode to be treated as a protected-mode general-protection excep- 
tion (#GP). The general-protection exception handler calls the virtual-8086 monitor, which can then emulate an 
8086-program interrupt handler or pass control back to the 8086 program’s handler, as described in Section 
19.3.1.2, “Handling an Interrupt or Exception With an 8086 Program Interrupt or Exception Handler”. 


Method 3 is enabled when the VME flag is set to 1, the IOPL value is less than 3, and the corresponding bit for the 
software interrupt in the software interrupt redirection bit map is set to 1. Here, the processor performs the same 
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operation as it does for method 2 software interrupt handling. If the corresponding bit for the software interrupt in 
the software interrupt redirection bit map is set to 0, the interrupt is handled using method 6 (see Section 19.3.3.5, 
“Method 6: Software Interrupt Handling”). 


19.3.3.3. Method 4: Software Interrupt Handling 


Method 4 handling is enabled when the VME flag is set to 1, the IOPL value is 3, and the bit for the interrupt vector 
in the redirection bit map is set to 1. Method 4 software interrupt handling allows method 1 style handling when the 
virtual mode extension is enabled; that is, the interrupt is directed to a protected-mode handler (see Section 
19.3.3.1, “Method 1: Software Interrupt Handling”). 


19.3.3.4 Method 5: Software Interrupt Handling 


Method 5 software interrupt handling provides a streamlined method of redirecting software interrupts (invoked 
with the INT rn instruction) that occur in virtual 8086 mode back to the 8086 program’s interrupt vector table and 
its interrupt handlers. Method 5 handling is enabled when the VME flag is set to 1, the IOPL value is 3, and the bit 
for the interrupt vector in the redirection bit map is set to 0. The processor performs the following actions to make 
an implicit call to the selected 8086 program interrupt handler: 


1. Pushes the low-order 16 bits of the EFLAGS register onto the stack. 


2. Pushes the current values of the CS and EIP registers onto the current stack. (Only the 16 least-significant bits 
of the EIP register are pushed and no stack switch occurs. ) 


Clears the IF flag in the EFLAGS register to disable interrupts. 
Clears the TF flag, in the EFLAGS register. 
Locates the 8086 program interrupt vector table at linear address O for the 8086-mode task. 


Oe 


Loads the CS and EIP registers with values from the interrupt vector table entry pointed to by the interrupt 
vector number. Only the 16 low-order bits of the EIP are loaded and the 16 high-order bits are set to 0. The 
interrupt vector table is assumed to be at linear address O of the current virtual-8086 task. 


7. Begins executing the selected interrupt handler. 


An IRET instruction at the end of the handler procedure reverses these steps to return program control to the inter- 
rupted 8086 program. 


Note that with method 5 handling, a mode switch from virtual-8086 mode to protected mode does not occur. The 
processor remains in virtual-8086 mode throughout the interrupt-handling operation. 


The method 5 handling actions are virtually identical to the actions the processor takes when handling software 
interrupts in real-address mode. The benefit of using method 5 handling to access the 8086 program handlers is 
that it avoids the overhead of methods 2 and 3 handling, which requires first going to the virtual-8086 monitor, 
then to the 8086 program handler, then back again to the virtual-8086 monitor, before returning to the interrupted 
8086 program (see Section 19.3.1.2, “Handling an Interrupt or Exception With an 8086 Program Interrupt or 
Exception Handler”). 


NOTE 


Methods 1 and 4 handling can handle a software interrupt in a virtual-8086 task with a regular 
protected-mode handler, but this approach requires all virtual-8086 tasks to use the same software 
interrupt handlers, which generally does not give sufficient latitude to the programs running in the 
virtual-8086 tasks, particularly MS-DOS programs. 


19.3.3.5 Method 6: Software Interrupt Handling 


Method 6 handling is enabled when the VME flag is set to 1, the IOPL value is less than 3, and the bit for the inter- 
rupt or exception vector in the redirection bit map is set to 0. With method 6 interrupt handling, software interrupts 
are handled in the same manner as was described for method 5 handling (see Section 19.3.3.4, “Method 5: Soft- 
ware Interrupt Handling”). 


Vol. 3B 19-19 


8086 EMULATION 


Method 6 differs from method 5 in that with the IOPL value set to less than 3, the VIF and VIP flags in the EFLAGS 
register are enabled, providing virtual interrupt support for handling class 2 maskable hardware interrupts (see 
Section 19.3.2, “Class 2—Maskable Hardware Interrupt Handling in Virtual-8086 Mode Using the Virtual Interrupt 
Mechanism”). These flags provide the virtual-8086 monitor with an efficient means of handling maskable hardware 
interrupts that occur during a virtual-8086 mode task. Also, because the IOPL value is less than 3 and the VIF flag 
is enabled, the information pushed on the stack by the processor when invoking the interrupt handler is slightly 
different between methods 5 and 6 (see Table 19-2). 


19.4 PROTECTED-MODE VIRTUAL INTERRUPTS 


The IA-32 processors (beginning with the Pentium processor) also support the VIF and VIP flags in the EFLAGS 
register in protected mode by setting the PVI (protected-mode virtual interrupt) flag in the CR4 register. Setting the 
PVI flag allows applications running at privilege level 3 to execute the CLI and STI instructions without causing a 
general-protection exception (#GP) or affecting hardware interrupts. 


When the PVI flag is set to 1, the CPL is 3, and the IOPL is less than 3, the STI and CLI instructions set and clear 
the VIF flag in the EFLAGS register, leaving IF unaffected. In this mode of operation, an application running in 
protected mode and at a CPL of 3 can inhibit interrupts in the same manner as is described in Section 19.3.2, “Class 
2—Maskable Hardware Interrupt Handling in Virtual-8086 Mode Using the Virtual Interrupt Mechanism”, for a 
virtual-8086 mode task. When the application executes the CLI instruction, the processor clears the VIF flag. If the 
processor receives a maskable hardware interrupt, the processor invokes the protected-mode interrupt handler. 
This handler checks the state of the VIF flag in the EFLAGS register. If the VIF flag is clear (indicating that the active 
task does not want to have interrupts handled now), the handler sets the VIP flag in the EFLAGS image on the stack 
and returns to the privilege-level 3 application, which continues program execution. When the application executes 
a STI instruction to set the VIF flag, the processor automatically invokes the general-protection exception handler, 
which can then handle the pending interrupt. After handing the pending interrupt, the handler typically sets the VIF 
flag and clears the VIP flag in the EFLAGS image on the stack and executes a return to the application program. The 
next time the processor receives a maskable hardware interrupt, the processor will handle it in the normal manner 
for interrupts received while the processor is operating at a CPL of 3. 


If the protected-mode virtual interrupt extension is enabled, CPL = 3, and the processor finds that both the VIF and 
VIP flags are set at the beginning of an instruction, a general-protection exception is generated. 


Because the protected-mode virtual interrupt extension changes only the treatment of EFLAGS.IF (by having CLI 
and STI update EFLAGS.VIF instead), it affects only the masking of maskable hardware interrupts (interrupt 
vectors 32 through 255). NMI interrupts and exceptions are handled in the normal manner. 


(When protected-mode virtual interrupts are disabled (that is, when the PVI flag in control register CR4 is set to 0, 
the CPL is less than 3, or the IOPL value is 3), then the CLI and STI instructions execute in a manner compatible 
with the Intel486 processor. That is, if the CPL is greater (less privileged) than the I/O privilege level (IOPL), a 
general-protection exception occurs. If the IOPL value is 3, CLI and STI clear or set the IF flag, respectively. ) 


PUSHF, POPF, IRET and INT are executed like in the Intel486 processor, regardless of whether protected-mode 
virtual interrupts are enabled. 


It is only possible to enter virtual-8086 mode through a task switch or the execution of an IRET instruction, and it 
is only possible to leave virtual-8086 mode by faulting to a protected-mode interrupt handler (typically the general- 
protection exception handler, which in turn calls the virtual 8086-mode monitor). In both cases, the EFLAGS 
register is saved and restored. This is not true, however, in protected mode when the PVI flag is set and the 
processor is not in virtual-8086 mode. Here, it is possible to call a procedure at a different privilege level, in which 
case the EFLAGS register is not saved or modified. However, the states of VIF and VIP flags are never examined by 
the processor when the CPL is not 3. 
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CHAPTER 20 
MIXING 16-BIT AND 32-BIT CODE 


Program modules written to run on IA-32 processors can be either 16-bit modules or 32-bit modules. Table 20-1 
shows the characteristic of 16-bit and 32-bit modules. 


Table 20-1. Characteristics of 16-Bit and 32-Bit Program Modules 


Characteristic 16-Bit Program Modules 32-Bit Program Modules 
Segment Size 0 to 64 KBytes 0 to 4 GBytes 
Operand Sizes 8 bits and 16 bits 8 bits and 32 bits 
Pointer Offset Size (Address Size) 16 bits 32 bits 
Stack Pointer Size 16 Bits 32 Bits 
Control Transfers Allowed to Code Segments | 16 Bits 32 Bits 
of This Size 


The IA-32 processors function most efficiently when executing 32-bit program modules. They can, however, also 
execute 16-bit program modules, in any of the following ways: 


° Inreal-address mode. 
¢ In virtual-8086 mode. 
¢ System management mode (SMM). 


e Asa protected-mode task, when the code, data, and stack segments for the task are all configured as a 16-bit 
segments. 


° By integrating 16-bit and 32-bit segments into a single protected-mode task. 
¢ By integrating 16-bit operations into 32-bit code segments. 


Real-address mode, virtual-8086 mode, and SMM are native 16-bit modes. A legacy program assembled and/or 
compiled to run on an Intel 8086 or Intel 286 processor should run in real-address mode or virtual-8086 mode 
without modification. Sixteen-bit program modules can also be written to run in real-address mode for handling 
system initialization or to run in SMM for handling system management functions. See Chapter 19, “8086 Emula- 
tion,” for detailed information on real-address mode and virtual-8086 mode; see Chapter 30, “System Manage- 
ment Mode,” for information on SMM. 


This chapter describes how to integrate 16-bit program modules with 32-bit program modules when operating in 
protected mode and how to mix 16-bit and 32-bit code within 32-bit code segments. 


20.1. DEFINING 16-BIT AND 32-BIT PROGRAM MODULES 


The following IA-32 architecture mechanisms are used to distinguish between and support 16-bit and 32-bit 
segments and operations: 


° The D (default operand and address size) flag in code-segment descriptors. 
° The B (default stack size) flag in stack-segment descriptors. 

® 16-bit and 32-bit call gates, interrupt gates, and trap gates. 

* Operand-size and address-size instruction prefixes. 

° 16-bit and 32-bit general-purpose registers. 


The D flag in a code-segment descriptor determines the default operand-size and address-size for the instructions 
of a code segment. (In real-address mode and virtual-8086 mode, which do not use segment descriptors, the 
default is 16 bits.) A code segment with its D flag set is a 32-bit segment; a code segment with its D flag clear is a 
16-bit segment. 
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The B flag in the stack-segment descriptor specifies the size of stack pointer (the 32-bit ESP register or the 16-bit 
SP register) used by the processor for implicit stack references. The B flag for all data descriptors also controls 
upper address range for expand down segments. 


When transferring program control to another code segment through a call gate, interrupt gate, or trap gate, the 
operand size used during the transfer is determined by the type of gate used (16-bit or 32-bit), (not by the D-flag 
or prefix of the transfer instruction). The gate type determines how return information is saved on the stack (or 
stacks). 


For most efficient and trouble-free operation of the processor, 32-bit programs or tasks should have the D flag in 
the code-segment descriptor and the B flag in the stack-segment descriptor set, and 16-bit programs or tasks 
should have these flags clear. Program control transfers from 16-bit segments to 32-bit segments (and vice versa) 
are handled most efficiently through call, interrupt, or trap gates. 


Instruction prefixes can be used to override the default operand size and address size of a code segment. These 
prefixes can be used in real-address mode as well as in protected mode and virtual-8086 mode. An operand-size or 
address-size prefix only changes the size for the duration of the instruction. 


20.2 MIXING 16-BIT AND 32-BIT OPERATIONS WITHIN A CODE SEGMENT 


The following two instruction prefixes allow mixing of 32-bit and 16-bit operations within one segment: 
¢ The operand-size prefix (66H) 
¢ The address-size prefix (67H) 


These prefixes reverse the default size selected by the D flag in the code-segment descriptor. For example, the 
processor can interpret the (MOV mem, reg) instruction in any of four ways: 


° Ina 32-bit code segment: 
— Moves 32 bits from a 32-bit register to memory using a 32-bit effective address. 


— If preceded by an operand-size prefix, moves 16 bits from a 16-bit register to memory using a 32-bit 
effective address. 


— If preceded by an address-size prefix, moves 32 bits from a 32-bit register to memory using a 16-bit 
effective address. 


— If preceded by both an address-size prefix and an operand-size prefix, moves 16 bits from a 16-bit register 
to memory using a 16-bit effective address. 


° Ina 16-bit code segment: 
— Moves 16 bits from a 16-bit register to memory using a 16-bit effective address. 


— If preceded by an operand-size prefix, moves 32 bits from a 32-bit register to memory using a 16-bit 
effective address. 


— If preceded by an address-size prefix, moves 16 bits from a 16-bit register to memory using a 32-bit 
effective address. 


— If preceded by both an address-size prefix and an operand-size prefix, moves 32 bits from a 32-bit register 
to memory using a 32-bit effective address. 


The previous examples show that any instruction can generate any combination of operand size and address size 
regardless of whether the instruction is in a 16- or 32-bit segment. The choice of the 16- or 32-bit default for a code 
segment is normally based on the following criteria: 


¢ Performance — Always use 32-bit code segments when possible. They run much faster than 16-bit code 
segments on P6 family processors, and somewhat faster on earlier IA-32 processors. 


°* The operating system the code segment will be running on — If the operating system is a 16-bit 
operating system, it may not support 32-bit program modules. 


° Mode of operation — If the code segment is being designed to run in real-address mode, virtual-8086 mode, 
or SMM, it must be a 16-bit code segment. 
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¢ Backward compatibility to earlier IA-32 processors — If a code segment must be able to run on an Intel 
8086 or Intel 286 processor, it must be a 16-bit code segment. 


20.3. SHARING DATA AMONG MIXED-SIZE CODE SEGMENTS 


Data segments can be accessed from both 16-bit and 32-bit code segments. When a data segment that is larger 
than 64 KBytes is to be shared among 16- and 32-bit code segments, the data that is to be accessed from the 16- 
bit code segments must be located within the first 64 KBytes of the data segment. The reason for this is that 16- 
bit pointers by definition can only point to the first 64 KBytes of a segment. 


A stack that spans less than 64 KBytes can be shared by both 16- and 32-bit code segments. This class of stacks 
includes: 


° Stacks in expand-up segments with the G (granularity) and B (big) flags in the stack-segment descriptor clear. 
¢ Stacks in expand-down segments with the G and B flags clear. 


¢ Stacks in expand-up segments with the G flag set and the B flag clear and where the stack is contained 
completely within the lower 64 KBytes. (Offsets greater than FFFFH can be used for data, other than the stack, 
which is not shared.) 


See Section 3.4.5, “Segment Descriptors,” for a description of the G and B flags and the expand-down stack type. 


The B flag cannot, in general, be used to change the size of stack used by a 16-bit code segment. This flag controls 
the size of the stack pointer only for implicit stack references such as those caused by interrupts, exceptions, and 
the PUSH, POP, CALL, and RET instructions. It does not control explicit stack references, such as accesses to 
parameters or local variables. A 16-bit code segment can use a 32-bit stack only if the code is modified so that all 
explicit references to the stack are preceded by the 32-bit address-size prefix, causing those references to use 32- 
bit addressing and explicit writes to the stack pointer are preceded by a 32-bit operand-size prefix. 


In 32-bit, expand-down segments, all offsets may be greater than 64 KBytes; therefore, 16-bit code cannot use 
this kind of stack segment unless the code segment is modified to use 32-bit addressing. 


20.4 TRANSFERRING CONTROL AMONG MIXED-SIZE CODE SEGMENTS 


There are three ways for a procedure in a 16-bit code segment to safely make a call to a 32-bit code segment: 
¢ Make the call through a 32-bit call gate. 


° Make a 16-bit call to a 32-bit interface procedure. The interface procedure then makes a 32-bit call to the 
intended destination. 


° Modify the 16-bit procedure, inserting an operand-size prefix before the call, to change it to a 32-bit call. 


Likewise, there are three ways for procedure in a 32-bit code segment to safely make a call to a 16-bit code 
segment: 


° Make the call through a 16-bit call gate. Here, the EIP value at the CALL instruction cannot exceed FFFFH. 


¢ Make a 32-bit call to a 16-bit interface procedure. The interface procedure then makes a 16-bit call to the 
intended destination. 


° Modify the 32-bit procedure, inserting an operand-size prefix before the call, changing it to a 16-bit call. Be 
certain that the return offset does not exceed FFFFH. 


These methods of transferring program control overcome the following architectural limitations imposed on calls 
between 16-bit and 32-bit code segments: 


¢ Pointers from 16-bit code segments (which by default can only be 16 bits) cannot be used to address data or 
code located beyond FFFFH in a 32-bit segment. 


¢ The operand-size attributes for a CALL and its companion RETURN instruction must be the same to maintain 
stack coherency. This is also true for implicit calls to interrupt and exception handlers and their companion IRET 
instructions. 

° A 32-bit parameters (particularly a pointer parameter) greater than FFFFH cannot be squeezed into a 16-bit 
parameter location on a stack. 
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¢ The size of the stack pointer (SP or ESP) changes when switching between 16-bit and 32-bit code segments. 
These limitations are discussed in greater detail in the following sections. 


20.4.1 Code-Segment Pointer Size 


For control-transfer instructions that use a pointer to identify the next instruction (that is, those that do not use 
gates), the operand-size attribute determines the size of the offset portion of the pointer. The implications of this 
rule are as follows: 


¢ AJMP, CALL, or RET instruction from a 32-bit segment to a 16-bit segment is always possible using a 32-bit 
operand size, providing the 32-bit pointer does not exceed FFFFH. 


° AJMP, CALL, or RET instruction from a 16-bit segment to a 32-bit segment cannot address a destination greater 
than FFFFH, unless the instruction is given an operand-size prefix. 


See Section 20.4.5, “Writing Interface Procedures,” for an interface procedure that can transfer program control 
from 16-bit segments to destinations in 32-bit segments beyond FFFFH. 


20.4.2 Stack Management for Control Transfer 


Because the stack is managed differently for 16-bit procedure calls than for 32-bit calls, the operand-size attribute 
of the RET instruction must match that of the CALL instruction (see Figure 20-1). On a 16-bit call, the processor 
pushes the contents of the 16-bit IP register and (for calls between privilege levels) the 16-bit SP register. The 
matching RET instruction must also use a 16-bit operand size to pop these 16-bit values from the stack into the 16- 
bit registers. 


A 32-bit CALL instruction pushes the contents of the 32-bit EIP register and (for inter-privilege-level calls) the 32- 
bit ESP register. Here, the matching RET instruction must use a 32-bit operand size to pop these 32-bit values from 
the stack into the 32-bit registers. If the two parts of a CALL/RET instruction pair do not have matching operand 
sizes, the stack will not be managed correctly and the values of the instruction pointer and stack pointer will not be 
restored to correct values. 
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Without Privilege Transition 


After 16-bit Call After 32-bit Call 


With Privilege Transition 


After 16-bit Call After 32-bit Call 
31 0 
ss SP 
Stack | PARM 2| PARM 1 
7 cs IP |<«—sp 
ESP 
Undefined 


Figure 20-1. Stack after Far 16- and 32-Bit Calls 


While executing 32-bit code, if a call is made to a 16-bit code segment which is at the same or a more privileged 
level (that is, the DPL of the called code segment is less than or equal to the CPL of the calling code segment) 
through a 16-bit call gate, then the upper 16-bits of the ESP register may be unreliable upon returning to the 32- 
bit code segment (that is, after executing a RET in the 16-bit code segment). 


When the CALL instruction and its matching RET instruction are in code segments that have D flags with the same 
values (that is, both are 32-bit code segments or both are 16-bit code segments), the default settings may be 
used. When the CALL instruction and its matching RET instruction are in segments which have different D-flag 
settings, an operand-size prefix must be used. 


20.4.2.1 Controlling the Operand-Size Attribute For a Call 


Three things can determine the operand-size of a call: 

¢ The D flag in the segment descriptor for the calling code segment. 

¢ An operand-size instruction prefix. 

° The type of call gate (16-bit or 32-bit), if a call is made through a call gate. 


When a call is made with a pointer (rather than a call gate), the D flag for the calling code segment determines the 
operand-size for the CALL instruction. This operand-size attribute can be overridden by prepending an operand- 
size prefix to the CALL instruction. So, for example, if the D flag for a code segment is set for 16 bits and the 
operand-size prefix is used with a CALL instruction, the processor will cause the information stored on the stack to 
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be stored in 32-bit format. If the call is to a 32-bit code segment, the instructions in that code segment will be able 
to read the stack coherently. Also, a RET instruction from the 32-bit code segment without an operand-size prefix 
will maintain stack coherency with the 16-bit code segment being returned to. 


When a CALL instruction references a call-gate descriptor, the type of call is determined by the type of call gate (16- 
bit or 32-bit). The offset to the destination in the code segment being called is taken from the gate descriptor; 
therefore, if a 32-bit call gate is used, a procedure in a 16-bit code segment can call a procedure located more than 
64 KBytes from the base of a 32-bit code segment, because a 32-bit call gate uses a 32-bit offset. 


Note that regardless of the operand size of the call and how it is determined, the size of the stack pointer used (SP 
or ESP) is always controlled by the B flag in the stack-segment descriptor currently in use (that is, when B is clear, 
SP is used, and when B is set, ESP is used). 


An unmodified 16-bit code segment that has run successfully on an 8086 processor or in real-mode on a later IA- 
32 architecture processor will have its D flag clear and will not use operand-size override prefixes. As a result, all 
CALL instructions in this code segment will use the 16-bit operand-size attribute. Procedures in these code 
segments can be modified to safely call procedures to 32-bit code segments in either of two ways: 


°®  Relink the CALL instruction to point to 32-bit call gates (see Section 20.4.2.2, “Passing Parameters With a 
Gate”). 


e =Add a 32-bit operand-size prefix to each CALL instruction. 


20.4.2.2 Passing Parameters With a Gate 


When referencing 32-bit gates with 16-bit procedures, it is important to consider the number of parameters passed 
in each procedure call. The count field of the gate descriptor specifies the size of the parameter string to copy from 
the current stack to the stack of a more privileged (numerically lower privilege level) procedure. The count field of 
a 16-bit gate specifies the number of 16-bit words to be copied, whereas the count field of a 32-bit gate specifies 
the number of 32-bit doublewords to be copied. The count field for a 32-bit gate must thus be half the size of the 
number of words being placed on the stack by a 16-bit procedure. Also, the 16-bit procedure must use an even 
number of words as parameters. 


20.4.3 ‘Interrupt Control Transfers 


A program-control transfer caused by an exception or interrupt is always carried out through an interrupt or trap 
gate (located in the IDT). Here, the type of the gate (16-bit or 32-bit) determines the operand-size attribute used 
in the implicit call to the exception or interrupt handler procedure in another code segment. 


A 32-bit interrupt or trap gate provides a safe interface to a 32-bit exception or interrupt handler when the excep- 
tion or interrupt occurs in either a 32-bit or a 16-bit code segment. It is sometimes impractical, however, to place 
exception or interrupt handlers in 16-bit code segments, because only 16-bit return addresses are saved on the 
stack. If an exception or interrupt occurs in a 32-bit code segment when the EIP was greater than FFFFH, the 16- 
bit handler procedure cannot provide the correct return address. 


20.4.4 Parameter Translation 


When segment offsets or pointers (which contain segment offsets) are passed as parameters between 16-bit and 
32-bit procedures, some translation is required. If a 32-bit procedure passes a pointer to data located beyond 64 
KBytes to a 16-bit procedure, the 16-bit procedure cannot use it. Except for this limitation, interface code can 
perform any format conversion between 32-bit and 16-bit pointers that may be needed. 


Parameters passed by value between 32-bit and 16-bit code also may require translation between 32-bit and 16- 
bit formats. The form of the translation is application-dependent. 


20.4.5 Writing Interface Procedures 


Placing interface code between 32-bit and 16-bit procedures can be the solution to the following interface prob- 
lems: 
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Allowing procedures in 16-bit code segments to call procedures with offsets greater than FFFFH in 32-bit code 
segments. 


Matching operand-size attributes between companion CALL and RET instructions. 


Translating parameters (data), including managing parameter strings with a variable count or an odd number 
of 16-bit words. 


The possible invalidation of the upper bits of the ESP register. 


The interface procedure is simplified where these rules are followed. 


1. 


2; 
3. 


The interface procedure must reside in a 32-bit code segment (the D flag for the code-segment descriptor is 
set). 


All procedures that may be called by 16-bit procedures must have offsets not greater than FFFFH. 
All return addresses saved by 16-bit procedures must have offsets not greater than FFFFH. 


The interface procedure becomes more complex if any of these rules are violated. For example, if a 16-bit proce- 
dure calls a 32-bit procedure with an entry point beyond FFFFH, the interface procedure will need to provide the 
offset to the entry point. The mapping between 16- and 32-bit addresses is only performed automatically when a 
call gate is used, because the gate descriptor for a call gate contains a 32-bit address. When a call gate is not used, 
the interface code must provide the 32-bit address. 


The structure of the interface procedure depends on the types of calls it is going to support, as follows: 


Calls from 16-bit procedures to 32-bit procedures — Calls to the interface procedure from a 16-bit code 
segment are made with 16-bit CALL instructions (by default, because the D flag for the calling code-segment 
descriptor is clear), and 16-bit operand-size prefixes are used with RET instructions to return from the interface 
procedure to the calling procedure. Calls from the interface procedure to 32-bit procedures are performed with 
32-bit CALL instructions (by default, because the D flag for the interface procedure’s code segment is set), and 
returns from the called procedures to the interface procedure are performed with 32-bit RET instructions (also 
by default). 


Calls from 32-bit procedures to 16-bit procedures — Calls to the interface procedure from a 32-bit code 
segment are made with 32-bit CALL instructions (by default), and returns to the calling procedure from the 
interface procedure are made with 32-bit RET instructions (also by default). Calls from the interface procedure 
to 16-bit procedures require the CALL instructions to have the operand-size prefixes, and returns from the 
called procedures to the interface procedure are performed with 16-bit RET instructions (by default). 
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CHAPTER 21 
ARCHITECTURE COMPATIBILITY 


Intel 64 and IA-32 processors are binary compatible. Compatibility means that, within limited constraints, 
programs that execute on previous generations of processors will produce identical results when executed on later 
processors. The compatibility constraints and any implementation differences between the Intel 64 and IA-32 
processors are described in this chapter. 


Each new processor has enhanced the software visible architecture from that found in earlier Intel 64 and IA-32 
processors. Those enhancements have been defined with consideration for compatibility with previous and future 
processors. This chapter also summarizes the compatibility considerations for those extensions. 


21.1 PROCESSOR FAMILIES AND CATEGORIES 


IA-32 processors are referred to in several different ways in this chapter, depending on the type of compatibility 
information being related, as described in the following: 


¢ IA-32 Processors — All the Intel processors based on the Intel IA-32 Architecture, which include the 
8086/88, Intel 286, Intel386, Intel486, Pentium, Pentium Pro, Pentium II, Pentium III, Pentium 4, and Intel 
Xeon processors. 


° 32-bit Processors — All the IA-32 processors that use a 32-bit architecture, which include the Intel386, 
Intel486, Pentium, Pentium Pro, Pentium II, Pentium Ill, Pentium 4, and Intel Xeon processors. 


* 16-bit Processors — All the IA-32 processors that use a 16-bit architecture, which include the 8086/88 and 
Intel 286 processors. 


° P6 Family Processors — All the IA-32 processors that are based on the P6 microarchitecture, which include 
the Pentium Pro, Pentium II, and Pentium Ill processors. 


* Pentium® 4 Processors — A family of IA-32 and Intel 64 processors that are based on the Intel NetBurst® 
microarchitecture. 


* Intel® Pentium® M Processors — A family of IA-32 processors that are based on the Intel Pentium M 
processor microarchitecture. 


* Intel® Core™ Duo and Solo Processors — Families of IA-32 processors that are based on an improved Intel 
Pentium M processor microarchitecture. 


* Intel® Xeon® Processors — A family of IA-32 and Intel 64 processors that are based on the Intel NetBurst 
microarchitecture. This family includes the Intel Xeon processor and the Intel Xeon processor MP based on the 
Intel NetBurst microarchitecture. Intel Xeon processors 3000, 3100, 3200, 3300, 3200, 5100, 5200, 5300, 
5400, 7200, 7300 series are based on Intel Core microarchitectures and support Intel 64 architecture. 


* Pentium® D Processors — A family of dual-core Intel 64 processors that provides two processor cores in a 
physical package. Each core is based on the Intel NetBurst microarchitecture. 


* Pentium® Processor Extreme Editions — A family of dual-core Intel 64 processors that provides two 
processor cores in a physical package. Each core is based on the Intel NetBurst microarchitecture and supports 
Intel Hyper-Threading Technology. 


* Intel® Core™ 2 Processor family— A family of Intel 64 processors that are based on the Intel Core microar- 
chitecture. Intel Pentium Dual-Core processors are also based on the Intel Core microarchitecture. 


* Intel® Atom™ Processors — A family of IA-32 and Intel 64 processors. 45 nm Intel Atom processors are 
based on the Intel Atom microarchitecture. 32 nm Intel Atom processors are based on newer microarchitec- 
tures including the Silvermont microarchitecture and the Airmont microarchitecture. Each generation of Intel 
Atom processors can be identified by the CPUID’s DisplayFamily_DisplayModel signature; see Table 2-1 “CPUID 
Signature Values of DisplayFamily_DisplayModel” in Chapter 2, “Model-Specific Registers (MSRs)” of the Inte/® 
64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 
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21.2 RESERVED BITS 


Throughout this manual, certain bits are marked as reserved in many register and memory layout descriptions. 
When bits are marked as undefined or reserved, it is essential for compatibility with future processors that software 
treat these bits as having a future, though unknown effect. Software should follow these guidelines in dealing with 
reserved bits: 


* Do not depend on the states of any reserved bits when testing the values of registers or memory locations that 
contain such bits. Mask out the reserved bits before testing. 


* Do not depend on the states of any reserved bits when storing them to memory or to a register. 
* Do not depend on the ability to retain information written into any reserved bits. 


¢ When loading a register, always load the reserved bits with the values indicated in the documentation, if any, or 
reload them with values previously read from the same register. 


Software written for existing IA-32 processor that handles reserved bits correctly will port to future IA-32 proces- 
sors without generating protection exceptions. 


21.3. ENABLING NEW FUNCTIONS AND MODES 


Most of the new control functions defined for the P6 family and Pentium processors are enabled by new mode flags 
in the control registers (primarily register CR4). This register is undefined for IA-32 processors earlier than the 
Pentium processor. Attempting to access this register with an Intel486 or earlier IA-32 processor results in an 
invalid-opcode exception (#UD). Consequently, programs that execute correctly on the Intel486 or earlier IA-32 
processor cannot erroneously enable these functions. Attempting to set a reserved bit in register CR4 to a value 
other than its original value results in a general-protection exception (#GP). So, programs that execute on the P6 
family and Pentium processors cannot erroneously enable functions that may be implemented in future IA-32 
processors. 


The P6 family and Pentium processors do not check for attempts to set reserved bits in model-specific registers; 
however these bits may be checked on more recent processors. It is the obligation of the software writer to enforce 
this discipline. These reserved bits may be used in future Intel processors. 


21.4 DETECTING THE PRESENCE OF NEW FEATURES THROUGH SOFTWARE 


Software can check for the presence of new architectural features and extensions in either of two ways: 


1. Test for the presence of the feature or extension. Software can test for the presence of new flags in the EFLAGS 
register and control registers. If these flags are reserved (meaning not present in the processor executing the 
test), an exception is generated. Likewise, software can attempt to execute a new instruction, which results in 
an invalid-opcode exception (#UD) being generated if it is not supported. 


2. Execute the CPUID instruction. The CPUID instruction (added to the IA-32 in the Pentium processor) indicates 
the presence of new features directly. 


See Chapter 20, “Processor Identification and Feature Determination,” in the Inte/® 64 and IA-32 Architectures 
Software Developer’s Manual, Volume 1, for detailed information on detecting new processor features and exten- 
sions. 


21.5 INTEL MMX TECHNOLOGY 


The Pentium processor with MMX technology introduced the MMX technology and a set of MMX instructions to the 
IA-32. The MMX instructions are described in Chapter 9, “Programming with Intel® MMX™ Technology,” in the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, and in the Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volumes 2A, 2B, 2C & 2D. The MMX technology and MMxX instructions are 
also included in the Pentium II, Pentium Ill, Pentium 4, and Intel Xeon processors. 
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21.6 STREAMING SIMD EXTENSIONS (SSE) 


The Streaming SIMD Extensions (SSE) were introduced in the Pentium III processor. The SSE extensions consist of 
a new set of instructions and a new set of registers. The new registers include the eight 128-bit XMM registers and 
the 32-bit MXCSR control and status register. These instructions and registers are designed to allow SIMD compu- 
tations to be made on single-precision floating-point numbers. Several of these new instructions also operate in the 
MMX registers. SSE instructions and registers are described in Section 10, “Programming with Streaming SIMD 
Extensions (SSE),” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, and in the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volumes 2A, 2B, 2C & 2D. 


21.7. STREAMING SIMD EXTENSIONS 2 (SSE2) 


The Streaming SIMD Extensions 2 (SSE2) were introduced in the Pentium 4 and Intel Xeon processors. They 
consist of a new set of instructions that operate on the XMM and MXCSR registers and perform SIMD operations on 
double-precision floating-point values and on integer values. Several of these new instructions also operate in the 
MMxX registers. SSE2 instructions and registers are described in Chapter 11, “Programming with Streaming SIMD 
Extensions 2 (SSE2),” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, and in the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volumes 2A, 2B, 2C & 2D. 


21.8 STREAMING SIMD EXTENSIONS 3 (SSE3) 


The Streaming SIMD Extensions 3 (SSE3) were introduced in Pentium 4 processors supporting Intel Hyper- 
Threading Technology and Intel Xeon processors. SSE3 extensions include 13 instructions. Ten of these 13 instruc- 
tions support the single instruction multiple data (SIMD) execution model used with SSE/SSE2 extensions. One 
SSE3 instruction accelerates x87 style programming for conversion to integer. The remaining two instructions 
(MONITOR and MWAIT) accelerate synchronization of threads. SSE3 instructions are described in Chapter 12, 
“Programming with Intel® SSE3, SSSE3,Intel® SSE4 AND Intel® AESNI,” in the Inte/® 64 and IA-32 Architec- 
tures Software Developer’s Manual, Volume 1, and in the Inte/® 64 and IA-32 Architectures Software Developer's 
Manual, Volumes 2A, 2B, 2C & 2D. 


21.9 ADDITIONAL STREAMING SIMD EXTENSIONS 


The Supplemental Streaming SIMD Extensions 3 (SSSE3) were introduced in the Intel Core 2 processor and Intel 
Xeon processor 5100 series. Streaming SIMD Extensions 4 provided 54 new instructions introduced in 45 nm Intel 
Xeon processors and Intel Core 2 processors. SSSE3, SSE4.1 and SSE4.2 instructions are described in Chapter 12, 
“Programming with Intel® SSE3, SSSE3,Intel® SSE4 AND Intel® AESNI,” in the Inte/® 64 and IA-32 Architec- 
tures Software Developer’s Manual, Volume 1, and in the Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volumes 2A, 2B, 2C & 2D. 


21.10 INTEL HYPER-THREADING TECHNOLOGY 


Intel Hyper-Threading Technology provides two logical processors that can execute two separate code streams 
(called threads) concurrently by using shared resources in a single processor core or in a physical package. 


This feature was introduced in the Intel Xeon processor MP and later steppings of the Intel Xeon processor, and 
Pentium 4 processors supporting Intel Hyper-Threading Technology. The feature is also found in the Pentium 
processor Extreme Edition. See also: Section 8.7, “Intel® Hyper-Threading Technology Architecture.” 


45 nm and 32 nm Intel Atom processors support Intel Hyper-Threading Technology. 


Intel Atom processors based on Silvermont and Airmont microarchitectures do not support Intel Hyper-Threading 
Technology. 
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21.11 MULTI-CORE TECHNOLOGY 


The Pentium D processor and Pentium processor Extreme Edition provide two processor cores in each physical 
processor package. See also: Section 8.5, “Intel® Hyper-Threading Technology and Intel® Multi-Core Technology, 
and Section 8.8, “Multi-Core Architecture.” Intel Core 2 Duo, Intel Pentium Dual-Core processors, Intel Xeon 
processors 3000, 3100, 5100, 5200 series provide two processor cores in each physical processor package. Intel 
Core 2 Extreme, Intel Core 2 Quad processors, Intel Xeon processors 3200, 3300, 5300, 5400, 7300 series provide 
two processor cores in each physical processor package. 


" 


21.12 SPECIFIC FEATURES OF DUAL-CORE PROCESSOR 
Dual-core processors may have some processor-specific features. Use CPUID feature flags to detect the availability 
features. Note the following: 


¢ CPUID Brand String — On Pentium processor Extreme Edition, the process will report the correct brand string 
only after the correct microcode updates are loaded. 


¢ Enhanced Intel SpeedStep Technology — This feature is supported in Pentium D processor but not in 
Pentium processor Extreme Edition. 


21.13 NEW INSTRUCTIONS IN THE PENTIUM AND LATER IA-32 PROCESSORS 


Table 21-1 identifies the instructions introduced into the IA-32 in the Pentium processor and later IA-32 processors. 


21.13.1 Instructions Added Prior to the Pentium Processor 
The following instructions were added in the Intel486 processor: 

¢ BSWAP (byte swap) instruction. 

¢ XADD (exchange and add) instruction. 

* CMPXCHG (compare and exchange) instruction. 

¢ INVD (invalidate cache) instruction. 

¢ WBINVD (write-back and invalidate cache) instruction. 

¢ INVLPG (invalidate TLB entry) instruction. 


Table 21-1. New Instruction in the Pentium Processor and Later IA-32 Processors 


Instruction CPUID Identification Bits Introduced In 
CMOVcc (conditional move) EDX, Bit 15 Pentium Pro processor 
FCMOVcc (floating-point conditional move) EDX, Bits O and 15 
FCOMI (floating-point compare and set EFLAGS) EDX, Bits O and 15 
RDPMC (read performance monitoring counters) EAX, Bits 8-11, set to GH; 
see Note 1 
UD2 (undefined) EAX, Bits 8-11, set to 6H 
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Table 21-1. New Instruction in the Pentium Processor and Later IA-32 Processors (Contd.) 


Instruction CPUID Identification Bits Introduced In 
CMPXCHG8B (compare and exchange 8 bytes) EDX, Bit 8 Pentium processor 
CPUID (CPU identification) None; see Note 2 
RDTSC (read time-stamp counter) EDX, Bit 4 
RDMSR (read model-specific register) EDX, Bit 5 
WRMSR (write model-specific register) EDX, Bit 5 
MMX Instructions EDX, Bit 23 


NOTES: 


1. The RDPMC instruction was introduced in the P6 family of processors and added to later model Pentium processors. This instruc- 
tion is model specific in nature and not architectural. 


2. The CPUID instruction is available in all Pentium and P6 family processors and in later models of the Intel486 processors. The ability 
to set and clear the ID flag (bit 21) in the EFLAGS register indicates the availability of the CPUID instruction. 


The following instructions were added in the Intel386 processor: 

e LSS, LFS, and LGS (load SS, FS, and GS registers). 

¢ Long-displacement conditional jumps. 

° ~=Single-bit instructions. 

¢ Bit scan instructions. 

* Double-shift instructions. 

¢ Byte set on condition instruction. 

° Move with sign/zero extension. 

¢ Generalized multiply instruction. 

¢ MOV to and from control registers. 

¢ MOV to and from test registers (now obsolete). 

¢ MOV to and from debug registers. 

¢ RSM (resume from SMM). This instruction was introduced in the Intel386 SL and Intel486 SL processors. 
The following instructions were added in the Intel 387 math coprocessor: 
°  FPREM1. 

¢ FUCOM, FUCOMP, and FUCOMPP. 


21.14 OBSOLETE INSTRUCTIONS 


The MOV to and from test registers instructions were removed from the Pentium processor and future IA-32 
processors. Execution of these instructions generates an invalid-opcode exception (#UD). 


21.15 UNDEFINED OPCODES 


All new instructions defined for Intel 64 and IA-32 processors use binary encodings that were reserved on earlier- 
generation processors. Generally, attempting to execute a reserved opcode results in an invalid-opcode (#UD) 
exception being generated. Consequently, programs that execute correctly on earlier-generation processors 
cannot erroneously execute these instructions and thereby produce unexpected results when executed on later 
Intel 64 processors. 


For compatibility with prior generations, there are a few reserved opcodes which do not result in a #UD but rather 
result in the same behavior as certain defined instructions. In the interest of standardization, it is recommended 
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that software not use the opcodes given below but instead use those defined in the Intel® 64 and IA-32 Architec- 
tures Software Developer’s Manual, Volumes 2A, 2B, 2C & 2D. 


The following items enumerate those reserved opcodes (referring in some cases to opcode groups as defined in 

Appendix A, “Opcode Map” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2D). 

¢ Immediate Group 1 - When not in 64-bit mode, instructions encoded with opcode 82H result in the behavior 
of the corresponding instructions encoded with opcode 80H. Depending on the Op/Reg field of the ModR/M 
Byte, these opcodes are the byte forms of ADD, OR, ADC, SBB, AND, SUB, XOR, CMP. (In 64-bit mode, these 
opcodes cause a #UD.) 

¢ Shift Group 2 /6 - Instructions encoded with opcodes COH, C1H, DOH, D1H, D2H, and D3H with value 110B in 
the Op/Reg field (/6) of the ModR/M Byte result in the behavior of the corresponding instructions with value 
100B in the Op/Reg field (/4). These are various forms of the SAL/SHL instruction. 

¢ Unary Group 3 /1 - Instructions encoded with opcodes F6H and F7H with value 001B in the Op/Reg field (/01) 
of the ModR/M Byte result in the behavior of the corresponding instructions with value O0OB in the Op/Reg field 
(/0). These are various forms of the TEST instruction. 

¢ Reserved NOP - Instructions encoded with the opcode OFODH or with the opcodes OF18H through OF1FH 
result in the behavior of the NOP (No Operation) instruction, except for those opcodes defined in the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volumes 2A, 2B, 2C & 2D. The opcodes not so defined 
are considered “Reserved NOP” and may be used for future instructions which have no defined impact on 
existing architectural state. These reserved NOP opcodes are decoded with a ModR/M byte and typical 
instruction prefix options but still result in the behavior of the NOP instruction. 

° x87 Opcodes - There are several groups of x87 opcodes which provide the same behavior as other x87 
instructions. See Section 21.18.9 for the complete list. 


There are a few reserved opcodes that provide unique behavior but do not provide capabilities that are not already 
available in the main instructions defined in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 
Volumes 2A, 2B, 2C & 2D. 


° D6H - When not in 64-bit mode SALC - Set AL to Cary flag. IF (CF=1), AL=FF, ELSE, AL=0 (#UD in 64-bit 
mode) 


° x87 Opcodes - There are a few x87 opcodes with subtly different behavior from existing x87 instructions. See 
Section 21.18.9 for details. 


21.16 NEW FLAGS IN THE EFLAGS REGISTER 


The section titled “EFLAGS Register” in Chapter 3, “Basic Execution Environment,” of the Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 1, shows the configuration of flags in the EFLAGS register for 
the P6 family processors. No new flags have been added to this register in the P6 family processors. The flags 
added to this register in the Pentium and Intel486 processors are described in the following sections. 


The following flags were added to the EFLAGS register in the Pentium processor: 
¢ VIF (virtual interrupt flag), bit 19. 

¢ VIP (virtual interrupt pending), bit 20. 

¢ ID (identification flag), bit 21. 

The AC flag (bit 18) was added to the EFLAGS register in the Intel486 processor. 


21.16.1 Using EFLAGS Flags to Distinguish Between 32-Bit IA-32 Processors 


The following bits in the EFLAGS register that can be used to differentiate between the 32-bit IA-32 processors: 


° Bit 18 (the AC flag) can be used to distinguish an Intel386 processor from the P6 family, Pentium, and Intel486 
processors. Since it is not implemented on the Intel386 processor, it will always be clear. 


¢ Bit 21 (the ID flag) indicates whether an application can execute the CPUID instruction. The ability to set and 
clear this bit indicates that the processor is a P6 family or Pentium processor. The CPUID instruction can then 
be used to determine which processor. 
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¢ Bits 19 (the VIF flag) and 20 (the VIP flag) will always be zero on processors that do not support virtual mode 
extensions, which includes all 32-bit processors prior to the Pentium processor. 


See Chapter 20, “Processor Identification and Feature Determination,” in the Inte/® 64 and IA-32 Architectures 
Software Developer’s Manual, Volume 1, for more information on identifying processors. 


21.17 STACK OPERATIONS AND USER SOFTWARE 


This section identifies the differences in stack implementation between the various IA-32 processors. 


21.17.1 PUSH SP 


The P6 family, Pentium, Intel486, Intel386, and Intel 286 processors push a different value on the stack for a PUSH 
SP instruction than the 8086 processor. The 32-bit processors push the value of the SP register before it is decre- 
mented as part of the push operation; the 8086 processor pushes the value of the SP register after it is decre- 
mented. If the value pushed is important, replace PUSH SP instructions with the following three instructions: 


PUSH BP 
MOV BP, SP 
XCHG BP, [BP] 


This code functions as the 8086 processor PUSH SP instruction on the P6 family, Pentium, Intel486, Intel386, and 
Intel 286 processors. 


21.17.2 EFLAGS Pushed on the Stack 


The setting of the stored values of bits 12 through 15 (which includes the IOPL field and the NT flag) in the EFLAGS 
register by the PUSHF instruction, by interrupts, and by exceptions is different with the 32-bit IA-32 processors 
than with the 8086 and Intel 286 processors. The differences are as follows: 


* 8086 processor—bits 12 through 15 are always set. 
¢ Intel 286 processor—bits 12 through 15 are always cleared in real-address mode. 


¢ 32-bit processors in real-address mode—bit 15 (reserved) is always cleared, and bits 12 through 14 have the 
last value loaded into them. 


21.18 X87 FPU 


This section addresses the issues that must be faced when porting floating-point software designed to run on 
earlier IA-32 processors and math coprocessors to a Pentium 4, Intel Xeon, P6 family, or Pentium processor with 
integrated x87 FPU. To software, a Pentium 4, Intel Xeon, or P6 family processor looks very much like a Pentium 
processor. Floating-point software which runs on a Pentium or Intel486 DX processor, or on an Intel486 SX 
processor/Intel 487 SX math coprocessor system or an Intel386 processor/Intel 387 math coprocessor system, 
will run with at most minor modifications on a Pentium 4, Intel Xeon, or P6 family processor. To port code directly 
from an Intel 286 processor/Intel 287 math coprocessor system or an Intel 8086 processor/8087 math copro- 
cessor system to a Pentium 4, Intel Xeon, P6 family, or Pentium processor, certain additional issues must be 
addressed. 


In the following sections, the term “32-bit x87 FPUs” refers to the P6 family, Pentium, and Intel486 DX processors, 
and to the Intel 487 SX and Intel 387 math coprocessors; the term “16-bit IA-32 math coprocessors” refers to the 
Intel 287 and 8087 math coprocessors. 
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21.18.1 Control Register CRO Flags 


The ET, NE, and MP flags in control register CRO control the interface between the integer unit of an IA-32 processor 
and either its internal x87 FPU or an external math coprocessor. The effect of these flags in the various IA-32 
processors are described in the following paragraphs. 


The ET (extension type) flag (bit 4 of the CRO register) is used in the Intel386 processor to indicate whether the 
math coprocessor in the system is an Intel 287 math coprocessor (flag is clear) or an Intel 387 DX math copro- 
cessor (flag is set). This bit is hardwired to 1 in the P6 family, Pentium, and Intel486 processors. 


The NE (Numeric Exception) flag (bit 5 of the CRO register) is used in the P6 family, Pentium, and Intel486 proces- 
sors to determine whether unmasked floating-point exceptions are reported internally through interrupt vector 16 
(flag is set) or externally through an external interrupt (flag is clear). On a hardware reset, the NE flag is initialized 
to 0, so software using the automatic internal error-reporting mechanism must set this flag to 1. This flag is nonex- 
istent on the Intel386 processor. 


As on the Intel 286 and Intel386 processors, the MP (monitor coprocessor) flag (bit 1 of register CRO) determines 
whether the WAIT/FWAIT instructions or waiting-type floating-point instructions trap when the context of the x87 
FPU is different from that of the currently-executing task. If the MP and TS flag are set, then a WAIT/FWAIT instruc- 
tion and waiting instructions will cause a device-not-available exception (interrupt vector 7). The MP flag is used on 
the Intel 286 and Intel386 processors to support the use of a WAIT/FWAIT instruction to wait on a device other 
than a math coprocessor. The device reports its status through the BUSY# pin. Since the P6 family, Pentium, and 
Intel486 processors do not have such a pin, the MP flag has no relevant use and should be set to 1 for normal oper- 
ation. 


21.18.2 x87 FPU Status Word 


This section identifies differences to the x87 FPU status word for the different IA-32 processors and math coproces- 
sors, the reason for the differences, and their impact on software. 


21.18.2.1 Condition Code Flags (CO through C3) 


The following information pertains to differences in the use of the condition code flags (CO through C3) located in 
bits 8, 9, 10, and 14 of the x87 FPU status word. 


After execution of an FINIT instruction or a hardware reset on a 32-bit x87 FPU, the condition code flags are set to 
0. The same operations on a 16-bit IA-32 math coprocessor leave these flags intact (they contain their prior value). 
This difference in operation has no impact on software and provides a consistent state after reset. 


Transcendental instruction results in the core range of the P6 family and Pentium processors may differ from the 
Intel486 DX processor and Intel 487 SX math coprocessor by 2 to 3 units in the last place (ulps)—(see “Transcen- 
dental Instruction Accuracy” in Chapter 8, “Programming with the x87 FPU,” of the Inte/® 64 and IA-32 Architec- 
tures Software Developer’s Manual, Volume 1). As a result, the value saved in the Ci flag may also differ. 


After an incomplete FPREM/FPREM1 instruction, the CO, Ci, and C3 flags are set to 0 on the 32-bit x87 FPUs. After 
the same operation on a 16-bit IA-32 math coprocessor, these flags are left intact. 


On the 32-bit x87 FPUs, the C2 flag serves as an incomplete flag for the FTAN instruction. On the 16-bit IA-32 math 
coprocessors, the C2 flag is undefined for the FPTAN instruction. This difference has no impact on software, 
because Intel 287 or 8087 programs do not check C2 after an FPTAN instruction. The use of this flag on later 
processors allows fast checking of operand range. 


21.18.2.2 Stack Fault Flag 


When unmasked stack overflow or underflow occurs on a 32-bit x87 FPU, the IE flag (bit 0) and the SF flag (bit 6) 
of the x87 FPU status word are set to indicate a stack fault and condition code flag C1 is set or cleared to indicate 
overflow or underflow, respectively. When unmasked stack overflow or underflow occurs on a 16-bit IA-32 math 
coprocessor, only the IE flag is set. Bit 6 is reserved on these processors. The addition of the SF flag on a 32-bit x87 
FPU has no impact on software. Existing exception handlers need not change, but may be upgraded to take advan- 
tage of the additional information. 
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21.18.3 x87 FPU Control Word 


Only affine closure is supported for infinity control on a 32-bit x87 FPU. The infinity control flag (bit 12 of the x87 
FPU control word) remains programmable on these processors, but has no effect. This change was made to 
conform to the IEEE Standard 754 for Binary Floating-Point Arithmetic. On a 16-bit IA-32 math coprocessor, both 
affine and projective closures are supported, as determined by the setting of bit 12. After a hardware reset, the 
default value of bit 12 is projective. Software that requires projective infinity arithmetic may give different results. 


21.18.4 x87 FPU Tag Word 


When loading the tag word of a 32-bit x87 FPU, using an FLDENV, FRSTOR, or FXRSTOR (Pentium III processor only) 
instruction, the processor examines the incoming tag and classifies the location only as empty or non-empty. Thus, 
tag values of 00, 01, and 10 are interpreted by the processor to indicate a non-empty location. The tag value of 11 
is interpreted by the processor to indicate an empty location. Subsequent operations on a non-empty register 
always examine the value in the register, not the value in its tag. The FSTENV, FSAVE, and FXSAVE (Pentium III 
processor only) instructions examine the non-empty registers and put the correct values in the tags before storing 
the tag word. 


The corresponding tag for a 16-bit IA-32 math coprocessor is checked before each register access to determine the 
class of operand in the register; the tag is updated after every change to a register so that the tag always reflects 
the most recent status of the register. Software can load a tag with a value that disagrees with the contents of a 
register (for example, the register contains a valid value, but the tag says special). Here, the 16-bit IA-32 math 
coprocessors honor the tag and do not examine the register. 


Software written to run on a 16-bit IA-32 math coprocessor may not operate correctly on a 16-bit x87 FPU, if it 
uses the FLDENV, FRSTOR, or FXRSTOR instructions to change tags to values (other than to empty) that are 
different from actual register contents. 


The encoding in the tag word for the 32-bit x87 FPUs for unsupported data formats (including pseudo-zero and 
unnormal) is special (10B), to comply with IEEE Standard 754. The encoding in the 16-bit IA-32 math coprocessors 
for pseudo-zero and unnormal is valid (OOB) and the encoding for other unsupported data formats is special (10B). 
Code that recognizes the pseudo-zero or unnormal format as valid must therefore be changed if it is ported to a 32- 
bit x87 FPU. 


21.18.5 Data Types 


This section discusses the differences of data types for the various x87 FPUs and math coprocessors. 


21.18.5.1 NaNs 


The 32-bit x87 FPUs distinguish between signaling NaNs (SNaNs) and quiet NaNs (QNaNs). These x87 FPUs only 
generate QNaNs and normally do not generate an exception upon encountering a QNaN. An invalid-operation 
exception (#1) is generated only upon encountering a SNaN, except for the FCOM, FIST, and FBSTP instructions, 
which also generates an invalid-operation exceptions for a QNaNs. This behavior matches IEEE Standard 754. 


The 16-bit IA-32 math coprocessors only generate one kind of NaN (the equivalent of a QNaN), but the raise an 
invalid-operation exception upon encountering any kind of NaN. 


When porting software written to run on a 16-bit IA-32 math coprocessor to a 32-bit x87 FPU, uninitialized memory 
locations that contain QNaNs should be changed to SNaNs to cause the x87 FPU or math coprocessor to fault when 
uninitialized memory locations are referenced. 


21.18.5.2 Pseudo-zero, Pseudo-NaN, Pseudo-infinity, and Unnormal Formats 


The 32-bit x87 FPUs neither generate nor support the pseudo-zero, pseudo-NaN, pseudo-infinity, and unnormal 
formats. Whenever they encounter them in an arithmetic operation, they raise an invalid-operation exception. The 
16-bit IA-32 math coprocessors define and support special handling for these formats. Support for these formats 
was dropped to conform with IEEE Standard 754 for Binary Floating-Point Arithmetic. 
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This change should not impact software ported from 16-bit IA-32 math coprocessors to 32-bit x87 FPUs. The 32- 
bit x87 FPUs do not generate these formats, and therefore will not encounter them unless software explicitly loads 
them in the data registers. The only affect may be in how software handles the tags in the tag word (see also: 
Section 21.18.4, “x87 FPU Tag Word”). 


21.18.6 Floating-Point Exceptions 


This section identifies the implementation differences in exception handling for floating-point instructions in the 
various x87 FPUs and math coprocessors. 


21.18.6.1 Denormal Operand Exception (#D) 


When the denormal operand exception is masked, the 32-bit x87 FPUs automatically normalize denormalized 
numbers when possible; whereas, the 16-bit IA-32 math coprocessors return a denormal result. A program written 
to run on a 16-bit IA-32 math coprocessor that uses the denormal exception solely to normalize denormalized 
operands is redundant when run on the 32-bit x87 FPUs. If such a program is run on 32-bit x87 FPUs, performance 
can be improved by masking the denormal exception. Floating-point programs run faster when the FPU performs 
normalization of denormalized operands. 


The denormal operand exception is not raised for transcendental instructions and the FXTRACT instruction on the 
16-bit IA-32 math coprocessors. This exception is raised for these instructions on the 32-bit x87 FPUs. The excep- 
tion handlers ported to these latter processors need to be changed only if the handlers gives special treatment to 
different opcodes. 


21.18.6.2 Numeric Overflow Exception (#0) 


On the 32-bit x87 FPUs, when the numeric overflow exception is masked and the rounding mode is set to chop 
(toward 0), the result is the largest positive or smallest negative number. The 16-bit IA-32 math coprocessors do 
not signal the overflow exception when the masked response is not ~; that is, they signal overflow only when the 
rounding control is not set to round to O. If rounding is set to chop (toward 0), the result is positive or negative ~. 
Under the most common rounding modes, this difference has no impact on existing software. 


If rounding is toward 0 (chop), a program on a 32-bit x87 FPU produces, under overflow conditions, a result that is 
different in the least significant bit of the significand, compared to the result on a 16-bit IA-32 math coprocessor. 
The reason for this difference is IEEE Standard 754 compatibility. 


When the overflow exception is not masked, the precision exception is flagged on the 32-bit x87 FPUs. When the 
result is stored in the stack, the significand is rounded according to the precision control (PC) field of the FPU 
control word or according to the opcode. On the 16-bit IA-32 math coprocessors, the precision exception is not 
flagged and the significand is not rounded. The impact on existing software is that if the result is stored on the 
stack, a program running on a 32-bit x87 FPU produces a different result under overflow conditions than on a 16- 
bit IA-32 math coprocessor. The difference is apparent only to the exception handler. This difference is for IEEE 
Standard 754 compatibility. 


21.18.6.3 Numeric Underflow Exception (#U) 


When the underflow exception is masked on the 32-bit x87 FPUs, the underflow exception is signaled when the 
result is tiny and inexact (see Section 4.9.1.5, “Numeric Underflow Exception (#U)” in Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 1). When the underflow exception is unmasked and the instruction 
is supposed to store the result on the stack, the significand is rounded to the appropriate precision (according to 
the PC flag in the FPU control word, for those instructions controlled by PC, otherwise to extended precision), after 
adjusting the exponent. 


21.18.6.4 Exception Precedence 


There is no difference in the precedence of the denormal-operand exception on the 32-bit x87 FPUs, whether it be 
masked or not. When the denormal-operand exception is not masked on the 16-bit IA-32 math coprocessors, it 
takes precedence over all other exceptions. This difference causes no impact on existing software, but some 


21-10 Vol. 3B 


ARCHITECTURE COMPATIBILITY 


unneeded normalization of denormalized operands is prevented on the Intel486 processor and Intel 387 math 
coprocessor. 


21.18.6.5 CS and EIP For FPU Exceptions 


On the Intel 32-bit x87 FPUs, the values from the CS and EIP registers saved for floating-point exceptions point to 
any prefixes that come before the floating-point instruction. On the 8087 math coprocessor, the saved CS and IP 
registers points to the floating-point instruction. 


21.18.6.6 FPU Error Signals 


The floating-point error signals to the P6 family, Pentium, and Intel486 processors do not pass through an interrupt 
controller; an INT# signal from an Intel 387, Intel 287 or 8087 math coprocessors does. If an 8086 processor uses 
another exception for the 8087 interrupt, both exception vectors should call the floating-point-error exception 
handler. Some instructions in a floating-point-error exception handler may need to be deleted if they use the inter- 
rupt controller. The P6 family, Pentium, and Intel486 processors have signals that, with the addition of external 
logic, support reporting for emulation of the interrupt mechanism used in many personal computers. 


On the P6 family, Pentium, and Intel486 processors, an undefined floating-point opcode will cause an invalid- 
opcode exception (#UD, interrupt vector 6). Undefined floating-point opcodes, like legal floating-point opcodes, 
cause a device not available exception (#NM, interrupt vector 7) when either the TS or EM flag in control register 
CRO is set. The P6 family, Pentium, and Intel486 processors do not check for floating-point error conditions on 
encountering an undefined floating-point opcode. 


21.18.6.7 Assertion of the FERR# Pin 


When using the MS-DOS compatibility mode for handing floating-point exceptions, the FERR# pin must be 
connected to an input to an external interrupt controller. An external interrupt is then generated when the FERR# 
output drives the input to the interrupt controller and the interrupt controller in turn drives the INTR pin on the 
processor. 


For the P6 family and Intel386 processors, an unmasked floating-point exception always causes the FERR# pin to 
be asserted upon completion of the instruction that caused the exception. For the Pentium and Intel486 proces- 
sors, an unmasked floating-point exception may cause the FERR# pin to be asserted either at the end of the 
instruction causing the exception or immediately before execution of the next floating-point instruction. (Note that 
the next floating-point instruction would not be executed until the pending unmasked exception has been 
handled.) See Appendix D, “Guidelines for Writing x87 FPU Extension Handlers,” in the Inte/® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 1, for a complete description of the required mechanism for 
handling floating-point exceptions using the MS-DOS compatibility mode. 


Using FERR# and IGNNE#¥ to handle floating-point exception is deprecated by modern operating systems; this 
approach also limits newer processors to operate with one logical processor active. 


21.18.6.8 Invalid Operation Exception On Denormals 


An invalid-operation exception is not generated on the 32-bit x87 FPUs upon encountering a denormal value when 
executing a FSQRT, FDIV, or FPREM instruction or upon conversion to BCD or to integer. The operation proceeds by 
first normalizing the value. On the 16-bit IA-32 math coprocessors, upon encountering this situation, the invalid- 
operation exception is generated. This difference has no impact on existing software. Software running on the 32- 
bit x87 FPUs continues to execute in cases where the 16-bit IA-32 math coprocessors trap. The reason for this 
change was to eliminate an exception from being raised. 


21.18.6.9 Alignment Check Exceptions (#AC) 


If alignment checking is enabled, a misaligned data operand on the P6 family, Pentium, and Intel486 processors 
causes an alignment check exception (#AC) when a program or procedure is running at privilege-level 3, except 
for the stack portion of the FSAVE/FNSAVE, FXSAVE, FRSTOR, and FXRSTOR instructions. 
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21.18.6.10 Segment Not Present Exception During FLDENV 


On the Intel486 processor, when a segment not present exception (#NP) occurs in the middle of an FLDENV 
instruction, it can happen that part of the environment is loaded and part not. In such cases, the FPU control word 
is left with a value of OO7FH. The P6 family and Pentium processors ensure the internal state is correct at all times 
by attempting to read the first and last bytes of the environment before updating the internal state. 


21.18.6.11 Device Not Available Exception (#NM) 


The device-not-available exception (#NM, interrupt 7) will occur in the P6 family, Pentium, and Intel486 processors 
as described in Section 2.5, “Control Registers,” Table 2-2, and Chapter 6, “Interrupt 7—Device Not Available 
Exception (#NM).” 


21.18.6.12 Coprocessor Segment Overrun Exception 


The coprocessor segment overrun exception (interrupt 9) does not occur in the P6 family, Pentium, and Intel486 
processors. In situations where the Intel 387 math coprocessor would cause an interrupt 9, the P6 family, Pentium, 
and Intel486 processors simply abort the instruction. To avoid undetected segment overruns, it is recommended 
that the floating-point save area be placed in the same page as the TSS. This placement will prevent the FPU envi- 
ronment from being lost if a page fault occurs during the execution of an FLDENV, FRSTOR, or FXRSTOR instruction 
while the operating system is performing a task switch. 


21.18.6.13 General Protection Exception (#GP) 


A general-protection exception (#GP, interrupt 13) occurs if the starting address of a floating-point operand falls 
outside a segment’s size. An exception handler should be included to report these programming errors. 


21.18.6.14 Floating-Point Error Exception (#MF) 


In real mode and protected mode (not including virtual-8086 mode), interrupt vector 16 must point to the floating- 
point exception handler. In virtual-8086 mode, the virtual-8086 monitor can be programmed to accommodate a 
different location of the interrupt vector for floating-point exceptions. 


21.18.7 Changes to Floating-Point Instructions 


This section identifies the differences in floating-point instructions for the various Intel FPU and math coprocessor 
architectures, the reason for the differences, and their impact on software. 


21.18.7.1 FDIV, FPREM, and FSQRT Instructions 


The 32-bit x87 FPUs support operations on denormalized operands and, when detected, an underflow exception 
can occur, for compatibility with the IEEE Standard 754. The 16-bit IA-32 math coprocessors do not operate on 
denormalized operands or return underflow results. Instead, they generate an invalid-operation exception when 
they detect an underflow condition. An existing underflow exception handler will require change only if it gives 
different treatment to different opcodes. Also, it is possible that fewer invalid-operation exceptions will occur. 


21.18.7.2 FSCALE Instruction 


With the 32-bit x87 FPUs, the range of the scaling operand is not restricted. If (0 < | ST(1) < 1), the scaling factor 
is O; therefore, ST(O0) remains unchanged. If the rounded result is not exact or if there was a loss of accuracy 
(masked underflow), the precision exception is signaled. With the 16-bit IA-32 math coprocessors, the range of the 
scaling operand is restricted. If (0 < | ST(1) | < 1), the result is undefined and no exception is signaled. The 
impact of this difference on exiting software is that different results are delivered on the 32-bit and 16-bit FPUs and 
math coprocessors when (0 < | ST(1) | < 1). 
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21.18.7.3_ FPREM1 Instruction 


The 32-bit x87 FPUs compute a partial remainder according to IEEE Standard 754. This instruction does not exist 
on the 16-bit IA-32 math coprocessors. The availability of the FPREM1 instruction has is no impact on existing soft- 
ware. 


21.18.7.4 FPREM Instruction 


On the 32-bit x87 FPUs, the condition code flags CO, C3, C1 in the status word correctly reflect the three low-order 
bits of the quotient following execution of the FPREM instruction. On the 16-bit IA-32 math coprocessors, the 
quotient bits are incorrect when performing a reduction of (64% + M) when (N > 1) and M is 1 or 2. This difference 
does not affect existing software; software that works around the bug should not be affected. 


21.18.7.5 FUCOM, FUCOMP, and FUCOMPP Instructions 


When executing the FUCOM, FUCOMP, and FUCOMPP instructions, the 32-bit x87 FPUs perform unordered compare 
according to IEEE Standard 754. These instructions do not exist on the 16-bit IA-32 math coprocessors. The avail- 
ability of these new instructions has no impact on existing software. 


21.18.7.6 FPTAN Instruction 


On the 32-bit x87 FPUs, the range of the operand for the FPTAN instruction is much less restricted (| ST(0) | < 2°) 
than on earlier math coprocessors. The instruction reduces the operand internally using an internal x/4 constant 
that is more accurate. The range of the operand is restricted to (| ST(0) | < 2/4) on the 16-bit IA-32 math copro- 
cessors; the operand must be reduced to this range using FPREM. This change has no impact on existing software. 
See also sections 8.3.8 and section 8.3.10 of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 1 for more information on the accuracy of the FPTAN instruction. 


21.18.7.7 Stack Overflow 


On the 32-bit x87 FPUs, if an FPU stack overflow occurs when the invalid-operation exception is masked, the FPU 
returns the real, integer, or BCD-integer indefinite value to the destination operand, depending on the instruction 
being executed. On the 16-bit IA-32 math coprocessors, the original operand remains unchanged following a stack 
overflow, but it is loaded into register ST(1). This difference has no impact on existing software. 


21.18.7.8 FSIN, FCOS, and FSINCOS Instructions 


On the 32-bit x87 FPUs, these instructions perform three common trigonometric functions. These instructions do 
not exist on the 16-bit IA-32 math coprocessors. The availability of these instructions has no impact on existing 
software, but using them provides a performance upgrade. See also sections 8.3.8 and section 8.3.10 of the Inte/® 
64 and IA-32 Architectures Software Developer’s Manual, Volume 1 for more information on the accuracy of the 
FSIN, FCOS, and FSINCOS instructions. 


21.18.7.9 FPATAN Instruction 


On the 32-bit x87 FPUs, the range of operands for the FPATAN instruction is unrestricted. On the 16-bit IA-32 math 
coprocessors, the absolute value of the operand in register ST(0) must be smaller than the absolute value of the 
operand in register ST(1). This difference has impact on existing software. 


21.18.7.10 F2XM1 Instruction 


The 32-bit x87 FPUs support a wider range of operands (-1 < ST (0) < + 1) for the F2XM1 instruction. The 
supported operand range for the 16-bit IA-32 math coprocessors is (0 < ST(0) <0.5). This difference has no impact 
on existing software. 
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21.18.7.11 FLD Instruction 


On the 32-bit x87 FPUs, when using the FLD instruction to load an extended-real value, a denormal-operand excep- 
tion is not generated because the instruction is not arithmetic. The 16-bit IA-32 math coprocessors do report a 
denormal-operand exception in this situation. This difference does not affect existing software. 


On the 32-bit x87 FPUs, loading a denormal value that is in single- or double-real format causes the value to be 
converted to extended-real format. Loading a denormal value on the 16-bit IA-32 math coprocessors causes the 
value to be converted to an unnormal. If the next instruction is FXTRACT or FXAM, the 32-bit x87 FPUs will give a 
different result than the 16-bit IA-32 math coprocessors. This change was made for IEEE Standard 754 compati- 
bility. 

On the 32-bit x87 FPUs, loading an SNaN that is in single- or double-real format causes the FPU to generate an 
invalid-operation exception. The 16-bit IA-32 math coprocessors do not raise an exception when loading a signaling 
NaN. The invalid-operation exception handler for 16-bit math coprocessor software needs to be updated to handle 
this condition when porting software to 32-bit FPUs. This change was made for IEEE Standard 754 compatibility. 


21.18.7.12 FXTRACT Instruction 


On the 32-bit x87 FPUs, if the operand is 0 for the FXTRACT instruction, the divide-by-zero exception is reported 
and -- is delivered to register ST(1). If the operand is +o, no exception is reported. If the operand is 0 on the 16- 
bit IA-32 math coprocessors, 0 is delivered to register ST(1) and no exception is reported. If the operand is +0, the 
invalid-operation exception is reported. These differences have no impact on existing software. Software usually 
bypasses 0 and ~. This change is due to the IEEE Standard 754 recommendation to fully support the “logb” func- 
tion. 


21.18.7.13 Load Constant Instructions 


On 32-bit x87 FPUs, rounding control is in effect for the load constant instructions. Rounding control is not in effect 
for the 16-bit IA-32 math coprocessors. Results for the FLDPI, FLDLN2, FLDLG2, and FLDL2E instructions are the 
same as for the 16-bit IA-32 math coprocessors when rounding control is set to round to nearest or round to +>. 
They are the same for the FLDL2T instruction when rounding control is set to round to nearest, round to --9, or 
round to zero. Results are different from the 16-bit IA-32 math coprocessors in the least significant bit of the 
mantissa if rounding control is set to round to --° or round to O for the FLDPI, FLDLN2, FLDLG2, and FLDL2E instruc- 
tions; they are different for the FLDL2T instruction if round to +c is specified. These changes were implemented for 
compatibility with IEEE Standard 754 for Floating-Point Arithmetic recommendations. 


21.18.7.14 FXAM Instruction 


With the 32-bit x87 FPUs, if the FPU encounters an empty register when executing the FXAM instruction, it not 
generate combinations of CO through C3 equal to 1101 or 1111. The 16-bit IA-32 math coprocessors may generate 
these combinations, among others. This difference has no impact on existing software; it provides a performance 
upgrade to provide repeatable results. 


21.18.7.15 FSAVE and FSTENV Instructions 


With the 32-bit x87 FPUs, the address of a memory operand pointer stored by FSAVE or FSTENV is undefined if the 
previous floating-point instruction did not refer to memory 


21.18.8 Transcendental Instructions 


The floating-point results of the P6 family and Pentium processors for transcendental instructions in the core range 
may differ from the Intel486 processors by about 2 or 3 ulps (see “Transcendental Instruction Accuracy” in Chapter 
8, “Programming with the x87 FPU,” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 

Volume 1). Condition code flag C1 of the status word may differ as a result. The exact threshold for underflow and 
overflow will vary by a few ulps. The P6 family and Pentium processors’ results will have a worst case error of less 
than 1 ulp when rounding to the nearest-even and less than 1.5 ulps when rounding in other modes. The transcen- 
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dental instructions are guaranteed to be monotonic, with respect to the input operands, throughout the domain 
supported by the instruction. 


Transcendental instructions may generate different results in the round-up flag (C1) on the 32-bit x87 FPUs. The 
round-up flag is undefined for these instructions on the 16-bit IA-32 math coprocessors. This difference has no 
impact on existing software. 


21.18.9 Obsolete Instructions and Undefined Opcodes 


The 8087 math coprocessor instructions FENI and FDISI, and the Intel 287 math coprocessor instruction FSETPM 
are treated as integer NOP instructions in the 32-bit x87 FPUs. If these opcodes are detected in the instruction 
stream, no specific operation is performed and no internal states are affected. FSETPM informed the Intel 287 math 
coprocessor that the processor was in protected mode. The 32-bit x87 FPUs handle all addressing and exception- 
pointer information, whether in protected mode or not. 


For compatibility with prior generations there are a few reserved x87 opcodes which do not result in an invalid- 
opcode (#UD) exception, but rather result in the same behavior as existing defined x87 instructions. In the interest 
of standardization, it is recommended that the opcodes defined in the Inte/® 64 and IA-32 Architectures Software 
Developer’s Manual, Volumes 2A, 2B, 2C & 2D be used for these operations for standardization. 


¢ DCDOH through DCD7H - Behaves the same as FCOM, D8DOH through D8D7H. 
¢ DCD8H through DCDFH - Behaves the same as FCOMP, D8D8H through D8DFH. 
® DOC8H through DOCFH - Behaves the same as FXCH, D9C8H through D9OCFH. 
¢ DEDOH through DED7H - Behaves the same as FCOMP, D8D8H through D8DFH. 
¢ DFDOH through DFD7H - Behaves the same as FSTP, DDD8H through DDDFH. 
¢ DFC8H through DFCFH - Behaves the same as FXCH, D9C8H through D9CFH. 
¢ DFD8H through DFDFH - Behaves the same as FSTP, DDD8H through DDDFH. 


There are a few reserved x87 opcodes which provide unique behavior but do not provide capabilities which are not 
already available in the main instructions defined in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volumes 2A, 2B, 2C & 2D. 


® D9DS8H through D9DFH - Behaves the same as FSTP (DDD8H through DDDFH) but won't cause a stack 
underflow exception. 


¢ DFCOH through DFC7H - Behaves the same as FFREE (DDCOH through DDD7H) with the addition of an x87 
stack POP. 


21.18.10 WAIT/FWAIT Prefix Differences 


On the Intel486 processor, when a WAIT/FWAIT instruction precedes a floating-point instruction (one which itself 
automatically synchronizes with the previous floating-point instruction), the WAIT/FWAIT instruction is treated as 
a no-op. Pending floating-point exceptions from a previous floating-point instruction are processed not on the 
WAIT/FWAIT instruction but on the floating-point instruction following the WAIT/FWAIT instruction. In such a case, 
the report of a floating-point exception may appear one instruction later on the Intel486 processor than on a P6 
family or Pentium FPU, or on Intel 387 math coprocessor. 


21.18.11 Operands Split Across Segments and/or Pages 


On the P6 family, Pentium, and Intel486 processor FPUs, when the first half of an operand to be written is inside a 
page or segment and the second half is outside, a memory fault can cause the first half to be stored but not the 
second half. In this situation, the Intel 387 math coprocessor stores nothing. 


21.18.12 FPU Instruction Synchronization 


On the 32-bit x87 FPUs, all floating-point instructions are automatically synchronized; that is, the processor auto- 
matically waits until the previous floating-point instruction has completed before completing the next floating-point 
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instruction. No explicit WAIT/FWAIT instructions are required to assure this synchronization. For the 8087 math 
coprocessors, explicit waits are required before each floating-point instruction to ensure synchronization. Although 
8087 programs having explicit WAIT instructions execute perfectly on the 32-bit IA-32 processors without reas- 
sembly, these WAIT instructions are unnecessary. 


21.19 SERIALIZING INSTRUCTIONS 


Certain instructions have been defined to serialize instruction execution to ensure that modifications to flags, regis- 
ters and memory are completed before the next instruction is executed (or in P6 family processor terminology 
“committed to machine state”). Because the P6 family processors use branch-prediction and out-of-order execu- 
tion techniques to improve performance, instruction execution is not generally serialized until the results of an 
executed instruction are committed to machine state (see Chapter 2, “Intel® 64 and IA-32 Architectures,” in the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1). 


As a result, at places in a program or task where it is critical to have execution completed for all previous instruc- 
tions before executing the next instruction (for example, at a branch, at the end of a procedure, or in multipro- 
cessor dependent code), it is useful to add a serializing instruction. See Section 8.3, “Serializing Instructions,” for 
more information on serializing instructions. 


21.20 FPU AND MATH COPROCESSOR INITIALIZATION 


Table 9-1 shows the states of the FPUs in the P6 family, Pentium, Intel486 processors and of the Intel 387 math 
coprocessor and Intel 287 coprocessor following a power-up, reset, or INIT, or following the execution of an 
FINIT/FNINIT instruction. The following is some additional compatibility information concerning the initialization of 
x87 FPUs and math coprocessors. 


21.20.1_ Intel® 387 and Intel® 287 Math Coprocessor Initialization 


Following an Intel386 processor reset, the processor identifies its coprocessor type (Intel® 287 or Intel® 387 DX 
math coprocessor) by sampling its ERROR# input some time after the falling edge of RESET# signal and before 
execution of the first floating-point instruction. The Intel 287 coprocessor keeps its ERROR# output in inactive 
state after hardware reset; the Intel 387 coprocessor keeps its ERROR# output in active state after hardware 
reset. 


Upon hardware reset or execution of the FINIT/FNINIT instruction, the Intel 387 math coprocessor signals an error 
condition. The P6 family, Pentium, and Intel486 processors, like the Intel 287 coprocessor, do not. 


21.20.2 Intel486 SX Processor and Intel 487 SX Math Coprocessor Initialization 


When initializing an Intel486 SX processor and an Intel 487 SX math coprocessor, the initialization routine should 
check the presence of the math coprocessor and should set the FPU related flags (EM, MP, and NE) in control 
register CRO accordingly (see Section 2.5, “Control Registers,” for a complete description of these flags). Table 21-2 
gives the recommended settings for these flags when the math coprocessor is present. The FSTCW instruction will 
give a value of FFFFH for the Intel486 SX microprocessor and 037FH for the Intel 487 SX math coprocessor. 
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Table 21-2. Recommended Values of the EM, MP, and NE Flags for Intel486 SX Microprocessor/Intel 487 SX Math 
Coprocessor System 


CRO Flags Intel486 SX Processor Only Intel 487 SX Math Coprocessor Present 
—M 1 0 
MP 0 1 
NE 1 0, for MS-DOS* systems 
1, for user-defined exception handler 


The EM and MP flags in register CRO are interpreted as shown in Table 21-3. 


Table 21-3. EM and MP Flag Interpretation 


EM MP Interpretation 

0 0 Floating-point instructions are passed to FPU; WAIT/FWAIT and other waiting-type instructions 
ignore TS. 

0 1 Floating-point instructions are passed to FPU; WAIT/FWAIT and other waiting-type instructions 
test TS. 

1 0 Floating-point instructions trap to emulator; WAIT/FWAIT and other waiting-type instructions 
ignore TS. 

1 1 Floating-point instructions trap to emulator; WAIT/FWAIT and other waiting-type instructions 
test TS. 


Following is an example code sequence to initialize the system and check for the presence of Intel486 SX 
processor/Intel 487 SX math coprocessor. 


fninit 

fstcw mem_loc 

mov ax, mem_loc 

cmp ax, 037fh 

jz Intel487_SX_Math_CoProcessor_present ;ax=037fh 
jmp Intel486_SX_microprocessor_present sax=ffffh 


If the Intel 487 SX math coprocessor is not present, the following code can be run to set the CRO register for the 
Intel486 SX processor. 


mov eax, crO 

and eax, fffffffdh ;make MP=0 

or eax, 0024h imake EM=1, NE=1 
mov crO, eax 


This initialization will cause any floating-point instruction to generate a device not available exception (#NH), inter- 
rupt 7. The software emulation will then take control to execute these instructions. This code is not required if an 
Intel 487 SX math coprocessor is present in the system. In that case, the typical initialization routine for the 
Intel486 SX microprocessor will be adequate. 


Also, when designing an Intel486 SX processor based system with an Intel 487 SX math coprocessor, timing loops 
should be independent of frequency and clocks per instruction. One way to attain this is to implement these loops 
in hardware and not in software (for example, BIOS). 


21.21 CONTROL REGISTERS 


The following sections identify the new control registers and control register flags and fields that were introduced 
to the 32-bit IA-32 in various processor families. See Figure 2-7 for the location of these flags and fields in the 
control registers. 
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The Pentium Ill processor introduced one new control flag in control register CR4: 
* OSXMMEXCPT (bit 10) — The OS will set this bit if it supports unmasked SIMD floating-point exceptions. 
The Pentium II processor introduced one new control flag in control register CR4: 


© OSFXSR (bit 9) — The OS supports saving and restoring the Pentium III processor state during context 
switches. 


The Pentium Pro processor introduced three new control flags in control register CR4: 


e PAE (bit 5) — Physical address extension. Enables paging mechanism to reference extended physical addresses 
when set; restricts physical addresses to 32 bits when clear (see also: Section 21.22.1.1, “Physical Memory 
Addressing Extension”). 


© PGE (bit 7) — Page global enable. Inhibits flushing of frequently-used or shared pages on CR3 writes (see also: 
Section 21.22.1.2, “Global Pages”). 


° PCE (bit 8) — Performance-monitoring counter enable. Enables execution of the RDPMC instruction at any 
protection level. 


The content of CR4 is OH following a hardware reset. 


Control register CR4 was introduced in the Pentium processor. This register contains flags that enable certain new 
extensions provided in the Pentium processor: 


¢ VME — Virtual-8086 mode extensions. Enables support for a virtual interrupt flag in virtual-8086 mode (see 
Section 19.3, “Interrupt and Exception Handling in Virtual-8086 Mode”). 


¢ PVI — Protected-mode virtual interrupts. Enables support for a virtual interrupt flag in protected mode (see 
Section 19.4, “Protected-Mode Virtual Interrupts”). 


¢ TSD — Time-stamp disable. Restricts the execution of the RDTSC instruction to procedures running at 
privileged level 0. 


¢ DE — Debugging extensions. Causes an undefined opcode (#UD) exception to be generated when debug 
registers DR4 and DRS are references for improved performance (see Section 21.23.3, ‘Debug Registers DR4 
and DR5"). 


° PSE — Page size extensions. Enables 4-MByte pages with 32-bit paging when set (see Section 4.3, “32-Bit 
Paging”). 


¢ MCE — Machine-check enable. Enables the machine-check exception, allowing exception handling for certain 
hardware error conditions (see Chapter 15, “Machine-Check Architecture”). 


The Intel486 processor introduced five new flags in control register CRO: 
¢ NE — Numeric error. Enables the normal mechanism for reporting floating-point numeric errors. 
¢ WP — Write protect. Write-protects read-only pages against supervisor-mode accesses. 


¢ AM — Alignment mask. Controls whether alignment checking is performed. Operates in conjunction with the AC 
(Alignment Check) flag. 


¢ NW — Not write-through. Enables write-throughs and cache invalidation cycles when clear and disables invali- 
dation cycles and write-throughs that hit in the cache when set. 


* CD — Cache disable. Enables the internal cache when clear and disables the cache when set. 
The Intel486 processor introduced two new flags in control register CR3: 


¢ PCD — Page-level cache disable. The state of this flag is driven on the PCD# pin during bus cycles that are not 
paged, such as interrupt acknowledge cycles, when paging is enabled. The PCD# pin is used to control caching 
in an external cache on a cycle-by-cycle basis. 


¢ PWT — Page-level write-through. The state of this flag is driven on the PWT# pin during bus cycles that are not 
paged, such as interrupt acknowledge cycles, when paging is enabled. The PWT# pin is used to control write 
through in an external cache on a cycle-by-cycle basis. 
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21.22 MEMORY MANAGEMENT FACILITIES 


The following sections describe the new memory management facilities available in the various IA-32 processors 
and some compatibility differences. 


21.22.1 New Memory Management Control Flags 


The Pentium Pro processor introduced three new memory management features: physical memory addressing 
extension, the global bit in page-table entries, and general support for larger page sizes. These features are only 
available when operating in protected mode. 


21.22.1.1 Physical Memory Addressing Extension 


The new PAE (physical address extension) flag in control register CR4, bit 5, may enable additional address lines 
on the processor, allowing extended physical addresses. This option can only be used when paging is enabled, 
using a new page-table mechanism provided to support the larger physical address range (see Section 4.1, “Paging 
Modes and Control Bits”). 


21.22.1.2 Global Pages 


The new PGE (page global enable) flag in control register CR4, bit 7, provides a mechanism for preventing 
frequently used pages from being flushed from the translation lookaside buffer (TLB). When this flag is set, 
frequently used pages (such as pages containing kernel procedures or common data tables) can be marked global 
by setting the global flag in a page-directory or page-table entry. 


On a task switch or a write to control register CR3 (which normally causes the TLBs to be flushed), the entries in 
the TLB marked global are not flushed. Marking pages global in this manner prevents unnecessary reloading of the 
TLB due to TLB misses on frequently used pages. See Section 4.10, “Caching Translation Information” for a 
detailed description of this mechanism. 


21.22.1.3 Larger Page Sizes 


The P6 family processors support large page sizes. For 32-bit paging, this facility is enabled with the PSE (page size 
extension) flag in control register CR4, bit 4. When this flag is set, the processor supports either 4-KByte or 4- 
MByte page sizes. PAE paging and 4-level paging! support 2-MByte pages regardless of the value of CR4.PSE (see 
Section 4.4, “PAE Paging” and Section 4.5, “4-Level Paging and 5-Level Paging”). See Chapter 4, “Paging,” for more 
information about large page sizes. 


21.22.2 CD and NW Cache Control Flags 


The CD and NW flags in control register CRO were introduced in the Intel486 processor. In the P6 family and 
Pentium processors, these flags are used to implement a writeback strategy for the data cache; in the Intel486 
processor, they implement a write-through strategy. See Table 11-5 for a comparison of these bits on the P6 family, 
Pentium, and Intel486 processors. For complete information on caching, see Chapter 11, ‘Memory Cache Control.” 


21.22.3 Descriptor Types and Contents 


Operating-system code that manages space in descriptor tables often contains an invalid value in the access-rights 
field of descriptor-table entries to identify unused entries. Access rights values of 80H and OOH remain invalid for 
the P6 family, Pentium, Intel486, Intel386, and Intel 286 processors. Other values that were invalid on the Intel 
286 processor may be valid on the 32-bit processors because uses for these bits have been defined. 


1. Earlier versions of this manual used the term “IA-32e paging” to identify 4-level paging. 
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21.22.4 Changes in Segment Descriptor Loads 


On the Intel386 processor, loading a segment descriptor always causes a locked read and write to set the accessed 
bit of the descriptor. On the P6 family, Pentium, and Intel486 processors, the locked read and write occur only if the 
bit is not already set. 


21.23 DEBUG FACILITIES 


The P6 family and Pentium processors include extensions to the Intel486 processor debugging support for break- 
points. To use the new breakpoint features, it is necessary to set the DE flag in control register CR4. 


21.23.1 Differences in Debug Register DR6 


It is not possible to write a 1 to reserved bit 12 in debug status register DR6 on the P6 family and Pentium proces- 
sors; however, it is possible to write a 1 in this bit on the Intel486 processor. See Table 9-1 for the different setting 
of this register following a power-up or hardware reset. 


21.23.2 Differences in Debug Register DR7 


The P6 family and Pentium processors determines the type of breakpoint access by the R/WO through R/W3 fields 
in debug control register DR7 as follows: 


00 Break on instruction execution only. 
01 Break on data writes only. 
10 Undefined if the DE flag in control register CR4 is cleared; break on I/O reads or writes but not instruction 


fetches if the DE flag in control register CR4 is set. 
11 Break on data reads or writes but not instruction fetches. 


On the P6 family and Pentium processors, reserved bits 11, 12, 14 and 15 are hard-wired to 0. On the Intel486 
processor, however, bit 12 can be set. See Table 9-1 for the different settings of this register following a power-up 
or hardware reset. 


21.23.3 Debug Registers DR4 and DR5 


Although the DR4 and DR5 registers are documented as reserved, previous generations of processors aliased refer- 
ences to these registers to debug registers DR6 and DR7, respectively. When debug extensions are not enabled 
(the DE flag in control register CR4 is cleared), the P6 family and Pentium processors remain compatible with 
existing software by allowing these aliased references. When debug extensions are enabled (the DE flag is set), 
attempts to reference registers DR4 or DRS will result in an invalid-opcode exception (#UD). 


21.24 RECOGNITION OF BREAKPOINTS 


For the Pentium processor, it is recommended that debuggers execute the LGDT instruction before returning to the 
program being debugged to ensure that breakpoints are detected. This operation does not need to be performed 
on the P6 family, Intel486, or Intel386 processors. 


The implementation of test registers on the Intel486 processor used for testing the cache and TLB has been rede- 
signed using MSRs on the P6 family and Pentium processors. (Note that MSRs used for this function are different 
on the P6 family and Pentium processors.) The MOV to and from test register instructions generate invalid-opcode 
exceptions (#UD) on the P6 family processors. 
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21.25 EXCEPTIONS AND/OR EXCEPTION CONDITIONS 


This section describes the new exceptions and exception conditions added to the 32-bit IA-32 processors and 
implementation differences in existing exception handling. See Chapter 6, “Interrupt and Exception Handling,” for 
a detailed description of the IA-32 exceptions. 


The Pentium Ill processor introduced new state with the XMM registers. Computations involving data in these regis- 
ters can produce exceptions. A new MXCSR control/status register is used to determine which exception or excep- 
tions have occurred. When an exception associated with the XMM registers occurs, an interrupt is generated. 


¢ SIMD floating-point exception (#XM, interrupt 19) — New exceptions associated with the SIMD floating-point 
registers and resulting computations. 


No new exceptions were added with the Pentium Pro and Pentium II processors. The set of available exceptions is 
the same as for the Pentium processor. However, the following exception condition was added to the IA-32 with the 
Pentium Pro processor: 


¢ Machine-check exception (#MC, interrupt 18) — New exception conditions. Many exception conditions have 
been added to the machine-check exception and a new architecture has been added for handling and reporting 
on hardware errors. See Chapter 15, “Machine-Check Architecture,” for a detailed description of the new 
conditions. 


The following exceptions and/or exception conditions were added to the IA-32 with the Pentium processor: 


¢ Machine-check exception (#MC, interrupt 18) — New exception. This exception reports parity and other 
hardware errors. It is a model-specific exception and may not be implemented or implemented differently in 
future processors. The MCE flag in control register CR4 enables the machine-check exception. When this bit is 
clear (which it is at reset), the processor inhibits generation of the machine-check exception. 


° General-protection exception (#GP, interrupt 13) — New exception condition added. An attempt to write a 1 to 
a reserved bit position of a special register causes a general-protection exception to be generated. 


° =Page-fault exception (#PF, interrupt 14) — New exception condition added. When a 1 is detected in any of the 
reserved bit positions of a page-table entry, page-directory entry, or page-directory pointer during address 
translation, a page-fault exception is generated. 


The following exception was added to the Intel486 processor: 


¢ = Alignment-check exception (#AC, interrupt 17) — New exception. Reports unaligned memory references when 
alignment checking is being performed. 


The following exceptions and/or exception conditions were added to the Intel386 processor: 
*  Divide-error exception (#DE, interrupt 0) 


— Change in exception handling. Divide-error exceptions on the Intel386 processors always leave the saved 
CS:IP value pointing to the instruction that failed. On the 8086 processor, the CS:IP value points to the next 
instruction. 


— Change in exception handling. The Intel386 processors can generate the largest negative number as a 
quotient for the IDIV instruction (80H and 8000H). The 8086 processor generates a divide-error exception 
instead. 


¢ Invalid-opcode exception (#UD, interrupt 6) — New exception condition added. Improper use of the LOCK 
instruction prefix can generate an invalid-opcode exception. 


° =Page-fault exception (#PF, interrupt 14) — New exception condition added. If paging is enabled in a 16-bit 
program, a page-fault exception can be generated as follows. Paging can be used in a system with 16-bit tasks 
if all tasks use the same page directory. Because there is no place in a 16-bit TSS to store the PDBR register, 
switching to a 16-bit task does not change the value of the PDBR register. Tasks ported from the Intel 286 
processor should be given 32-bit TSSs so they can make full use of paging. 


° General-protection exception (#GP, interrupt 13) — New exception condition added. The Intel386 processor 
sets a limit of 15 bytes on instruction length. The only way to violate this limit is by putting redundant prefixes 
before an instruction. A general-protection exception is generated if the limit on instruction length is violated. 
The 8086 processor has no instruction length limit. 
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21.25.1 Machine-Check Architecture 


The Pentium Pro processor introduced a new architecture to the IA-32 for handling and reporting on machine-check 
exceptions. This machine-check architecture (described in detail in Chapter 15, “Machine-Check Architecture”) 
greatly expands the ability of the processor to report on internal hardware errors. 


21.25.2 Priority of Exceptions 

The priority of exceptions are broken down into several major categories: 
Traps on the previous instruction 

External interrupts 


Faults on fetching the next instruction 


PN 


Faults in decoding the next instruction 
5. Faults on executing an instruction 


There are no changes in the priority of these major categories between the different processors, however, excep- 
tions within these categories are implementation dependent and may change from processor to processor. 


21.25.3 Exception Conditions of Legacy SIMD Instructions Operating on MMX Registers 


MMX instructions and a subset of SSE, SSE2, SSSE3 instructions operate on MMX registers. The exception condi- 
tions of these instructions are described in the following tables. 


Table 21-4. Exception Conditions for Legacy SIMD/MMX Instructions with FP Exception and 16-Byte Alignment 
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Table 21-5. Exception Conditions for Legacy SIMD/MMX Instructions with XMM and FP Exception 
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#MF X X X X | If there is a pending X87 FPU exception 
#NM X X X X | If CRO.TS[bit 3]=1 
X For an illegal address in the SS segment 
Stack, SS(0) ; a ; 
X | lfamemory address referencing the SS segment is in a non-canonical form 
X For an illegal memory operand effective address in the CS, DS, ES, FS or GS seg- 
ments. 
General Protection, a ; 
#GP(0) X_ | If the memory address is in a non-canonical form. 
X X If any part of the operand lies outside the effective address space from O to 


FFFFH 


#PF(fault-code) 


Alignment Check 
#AC(0) 


X X X | Fora page fault 


If alignment checking is enabled and an unaligned memory reference is made while 
the current privilege level is 3. 


SIMD Floating-point 
Exception, #XM 


X X X X_| If an unmasked SIMD floating-point exception and CR4.0SXMMEXCPTI[bit 10] = 1 


Applicable Instruc- 
tions 


CVTPI2PS, CVTPS2PI, CVTTPS2PI 
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Table 21-6. Exception Conditions for Legacy SIMD/MMxX Instructions with XMM and without FP Exception 


tions 


Bg 52 
= Ss u05| & 
. oO t =] ya) i 
Exception tH) 3 |tol| y Cause of Exception 
a 3 |a2) © 
Ess 
> |av 
If CRO.EM[bit 2] = 1. 
a . . > If CR4.0SFXSR[bit 9] = 0. 
Invalid Opcode, #UD [Ty 7 x | x | ifpreceded by a LOCK prefix (FOH) 
X X X X_ | If any corresponding CPUID feature flag is ‘O’ 
#MF! X X X X_ | If there is a pending X87 FPU exception 
#NM X X X X_ | If CRO.TS[bit 3]=1 
X For an illegal address in the SS segment 
Stack, SS(0) : a : 
X | Ifamemory address referencing the SS segment is in a non-canonical form 
X For an illegal memory operand effective address in the CS, DS, ES, FS or GS seg- 
ments. 
iG, Rroteruan, X_ | If the memory address is in a non-canonical form. 
If any part of the operand lies outside the effective address space from O to 
X X 
FFFFH 
#PF(fault-code) X X X_ | For a page fault 
Alignment Check X X X If alignment checking is enabled and an unaligned memory reference is made 
#AC(0) while the current privilege level is 3. 
Applicable Instruc- CVTPI2PD 


NOTES: 


1. Applies to “CVTPIZPD xmm, mm” but not “CVTPIZPD xmm, m64". 


21-24 Vol. 3B 


ARCHITECTURE COMPATIBILITY 


Table 21-7. Exception Conditions for SIMD/MMX Instructions with Memory Reference 


a=) 
8 |se 
= S oo] £ 
. oO T os y=) i 
Exception a 3 ico] } Cause of Exception 
“| 2/82 ¢ 
5/28 

X X X X | If CRO.EM[bit 2] = 1. 
Invalid Opcode, #UD X X X X | If preceded by a LOCK prefix (FOH) 

X X X X_ | If any corresponding CPUID feature flag is ‘O’ 
#MF X X X X_ | If there is a pending X87 FPU exception 
#NM X X X X | If CRO.TS[bit 3]=1 

X For an illegal address in the SS segment 
Stack, SS(0) - = : 
X | Ifamemory address referencing the SS segment is in a non-canonical form 
X For an illegal memory operand effective address in the CS, DS, ES, FS or GS seg- 

General Protection, ments, 
#GP(0) X_| If the memory address is in a non-canonical form. 

X X If any part of the operand lies outside the effective address space from O to FFFFH 
#PF(fault-code) X X X_ | For a page fault 
Alignment Check X X X If alignment checking is enabled and an unaligned memory reference is made while 


#AC(0) 


the current privilege level is 3. 


Applicable Instruc- 
tions 


PABSB, PABSD, PABSW, PACKSSWB, PACKSSDW, PACKUSWB, PADDB, PADDD, PADDQ, PADDW, PADDSB, 
PADDSW, PADDUSB, PADDUSW, PALIGNR, PAND, PANDN, PAVGB, PAVGW, PCMPEQB, PCMPEQD, PCMPEQW, 
PCMPGTB, PCMPGTD, PCMPGTW, PHADDD, PHADDW, PHADDSW, PHSUBD, PHSUBW, PHSUBSW, PINSRW, 
PMADDUBSW, PMADDWD, PMAXSW, PMAXUB, PMINSW, PMINUB, PMULHRSW, PMULHUW, PMULHW, PMULLW, 
PMULUDQ, PSADBW, PSHUFB, PSHUFW, PSIGNB PSIGND PSIGNW, PSLLW, PSLLD, PSLLQ, PSRAD, PSRAW, 
PSRLW, PSRLD, PSRLQ, PSUBB, PSUBD, PSUBQ, PSUBW, PSUBSB, PSUBSW, PSUBUSB, PSUBUSW, 
PUNPCKHBW, PUNPCKHWD, PUNPCKHDQ, PUNPCKLBW, PUNPCKLWD, PUNPCKLDQ, PXOR 


Vol. 3B 21-25 


ARCHITECTURE COMPATIBILITY 


Table 21-8. Exception Conditions for Legacy SIMD/MMX Instructions without FP Exception 


a~) 
eS 
= oO 05| & 
. oO 7 vos] 2 i 
Exception 9\/sel/oa) y Cause of Exception 
©; s |/ao2flo 
Eos 
> av 
If CRO.EM[bit 2] = 1. 
X X X X 
If ModR/M.mod # 11b! 
Invalid Opcode, #UD ; 
P X X X X_ | If preceded by a LOCK prefix (FOH) 
X X X X_| If any corresponding CPUID feature flag is ‘O’ 
#MF X X X X | If there is a pending X87 FPU exception 
#NM X X X X | If CRO.TS[bit 3]=1 
X For an illegal address in the SS segment 
Stack, SS(0) - _ ; 
X | lfamemory address referencing the SS segment is in a non-canonical form 
For an illegal memory operand effective address in the CS, DS, ES, FS or GS seg- 
: ments. 
If the destination operand is in a non-writable segment.2 
#0P(0) If the DS, ES, FS, or GS register contains a NULL segment selector.2 
X_| If the memory address is in a non-canonical form. 
X X If any part of the operand lies outside the effective address space from O to FFFFH 
#PF(fault-code) X X X | For a page fault 
#AC(0) X X X If alignment checking is enabled and an unaligned memory reference is made while 
the current privilege level is 3. 


tions 


Applicable Instruc- MASKMOVQ, MOVNTQ, “MOVQ (mmreg)" 


NOTES: 
1. Applies to MASKMOVQ only. 


2. Applies to MASKMOVQ and MOVQ (mmreg) only. 


3. Applies to MASKMOVOQ only. 


21-26 Vol. 3B 


ARCHITECTURE COMPATIBILITY 


Table 21-9. Exception Conditions for Legacy SIMD/MMX Instructions without Memory Reference 


8 52 
_ S oO] £ 
. oO 7 os i=) i 
Exception a 3 iD o] ¥} Cause of Exception 
“| 2 /#e2 ¢ 
5 |28 
X X X X | If CRO.EM[bit 2] = 1. 
Invalid Opcode, #UD X X X X_| If preceded by a LOCK prefix (FOH) 
X X X X_ | If any corresponding CPUID feature flag is ‘O’ 
#MF X X X X | If there is a pending X87 FPU exception 
#NM X X_ | If CRO.TS[bit 3]=1 
Applicable Instruc- PEXTRW, PMOVMSKB 
tions 


21.26 INTERRUPTS 


The following differences in handling interrupts are found among the IA-32 
processors. 


21.26.1 Interrupt Propagation Delay 


External hardware interrupts may be recognized on different instruction boundaries on the P6 family, Pentium, 
Intel486, and Intel386 processors, due to the superscaler designs of the P6 family and Pentium processors. There- 
fore, the EIP pushed onto the stack when servicing an interrupt may be different for the P6 family, Pentium, 
Intel486, and Intel386 processors. 


21.26.2 NMI Interrupts 


After an NMI interrupt is recognized by the P6 family, Pentium, Intel486, Intel386, and Intel 286 processors, the 
NMI interrupt is masked until the first IRET instruction is executed, unlike the 8086 processor. 


21.26.3  IDT Limit 


The LIDT instruction can be used to set a limit on the size of the IDT. A double-fault exception (#DF) is generated 
if an interrupt or exception attempts to read a vector beyond the limit. Shutdown then occurs on the 32-bit IA-32 
processors if the double-fault handler vector is beyond the limit. (The 8086 processor does not have a shutdown 
mode nor a limit.) 


21.27 ADVANCED PROGRAMMABLE INTERRUPT CONTROLLER (APIC) 


The Advanced Programmable Interrupt Controller (APIC), referred to in this book as the local APIC, was intro- 
duced into the IA-32 processors with the Pentium processor (beginning with the 735/90 and 815/100 models) and 
is included in the Pentium 4, Intel Xeon, and P6 family processors. The features and functions of the local APIC are 
derived from the Intel 82489DX external APIC, which was used with the Intel486 and early Pentium processors. 
Additional refinements of the local APIC architecture were incorporated in the Pentium 4 and Intel Xeon processors. 
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21.27.1 Software Visible Differences Between the Local APIC and the 82489DX 


The following features in the local APIC features differ from those found in the 82489DX external APIC: 


¢ When the local APIC is disabled by clearing the APIC software enable/disable flag in the spurious-interrupt 
vector MSR, the state of its internal registers are unaffected, except that the mask bits in the LVT are all set to 
block local interrupts to the processor. Also, the local APIC ceases accepting IPIs except for INIT, SMI, NMI, and 
start-up IPIs. In the 82489DX, when the local unit is disabled, all the internal registers including the IRR, ISR 
and TMR are cleared and the mask bits in the LVT are set. In this state, the 82489DxX local unit will accept only 
the reset deassert message. 


¢ Inthe local APIC, NMI and INIT (except for INIT deassert) are always treated as edge triggered interrupts, even 
if programmed otherwise. In the 82489Dx, these interrupts are always level triggered. 


¢ Inthe local APIC, IPIs generated through the ICR are always treated as edge triggered (except INIT Deassert). 
In the 82489DX, the ICR can be used to generate either edge or level triggered IPIs. 


¢ In the local APIC, the logical destination register supports 8 bits; in the 82489DxX, it supports 32 bits. 
¢ In the local APIC, the APIC ID register is 4 bits wide; in the 82489Dx, it is 8 bits wide. 


¢ The remote read delivery mode provided in the 82489DX and local APIC for Pentium processors is not 
supported in the local APIC in the Pentium 4, Intel Xeon, and P6 family processors. 


° For the 82489Dx, in the lowest priority delivery mode, all the target local APICs specified by the destination 
field participate in the lowest priority arbitration. For the local APIC, only those local APICs which have free 
interrupt slots will participate in the lowest priority arbitration. 


21.27.2 New Features Incorporated in the Local APIC for the P6 Family and Pentium 
Processors 


The local APIC in the Pentium and P6 family processors have the following new features not found in the 82489DX 
external APIC. 


* Cluster addressing is supported in logical destination mode. 

® Focus processor checking can be enabled/disabled. 

¢ Interrupt input signal polarity can be programmed for the LINTO and LINT1 pins. 
¢ An SMI IPI is supported through the ICR and I/O redirection table. 

e Anerror status register is incorporated into the LVT to log and report APIC errors. 


In the P6 family processors, the local APIC incorporates an additional LVT register to handle performance moni- 
toring counter interrupts. 


21.27.3 New Features Incorporated in the Local APIC of the Pentium 4 and Intel Xeon 
Processors 


The local APIC in the Pentium 4 and Intel Xeon processors has the following new features not found in the P6 family 
and Pentium processors and in the 82489Dx. 


¢ The local APIC ID is extended to 8 bits. 

¢ Anthermal sensor register is incorporated into the LVT to handle thermal sensor interrupts. 

¢ The the ability to deliver lowest-priority interrupts to a focus processor is no longer supported. 
° ©The flat cluster logical destination mode is not supported. 


21.28 TASK SWITCHING AND TSS 


This section identifies the implementation differences of task switching, additions to the TSS and the handling of 
TSSs and TSS segment selectors. 
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21.28.1 P6 Family and Pentium Processor TSS 


When the virtual mode extensions are enabled (by setting the VME flag in control register CR4), the TSS in the P6 
family and Pentium processors contain an interrupt redirection bit map, which is used in virtual-8086 mode to redi- 
rect interrupts back to an 8086 program. 


21.28.2 TSS Selector Writes 


During task state saves, the Intel486 processor writes 2-byte segment selectors into a 32-bit TSS, leaving the 
upper 16 bits undefined. For performance reasons, the P6 family and Pentium processors write 4-byte segment 
selectors into the TSS, with the upper 2 bytes being 0. For compatibility reasons, code should not depend on the 
value of the upper 16 bits of the selector in the TSS. 


21.28.3 Order of Reads/Writes to the TSS 


The order of reads and writes into the TSS is processor dependent. The P6 family and Pentium processors may 
generate different page-fault addresses in control register CR2 in the same TSS area than the Intel486 and 
Intel386 processors, if a TSS crosses a page boundary (which is not recommended). 


21.28.4 Using A 16-Bit TSS with 32-Bit Constructs 


Task switches using 16-bit TSSs should be used only for pure 16-bit code. Any new code written using 32-bit 
constructs (operands, addressing, or the upper word of the EFLAGS register) should use only 32-bit TSSs. This is 
due to the fact that the 32-bit processors do not save the upper 16 bits of EFLAGS to a 16-bit TSS. A task switch 
back to a 16-bit task that was executing in virtual mode will never re-enable the virtual mode, as this flag was not 
saved in the upper half of the EFLAGS value in the TSS. Therefore, it is strongly recommended that any code using 
32-bit constructs use a 32-bit TSS to ensure correct behavior in a multitasking environment. 


21.28.5 Differences in I/O Map Base Addresses 


The Intel486 processor considers the TSS segment to be a 16-bit segment and wraps around the 64K boundary. 
Any I/O accesses check for permission to access this I/O address at the I/O base address plus the I/O offset. If the 
I/O map base address exceeds the specified limit of ODFFFH, an I/O access will wrap around and obtain the permis- 
sion for the I/O address at an incorrect location within the TSS. A TSS limit violation does not occur in this situation 
on the Intel486 processor. However, the P6 family and Pentium processors consider the TSS to be a 32-bit segment 
and a limit violation occurs when the I/O base address plus the I/O offset is greater than the TSS limit. By following 
the recommended specification for the I/O base address to be less than ODFFFH, the Intel486 processor will not 
wrap around and access incorrect locations within the TSS for I/O port validation and the P6 family and Pentium 
processors will not experience general-protection exceptions (#GP). Figure 21-1 demonstrates the different areas 
accessed by the Intel486 and the P6 family and Pentium processors. 
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Intel486 Processor P6 family and Pentium Processors 


FFFFH + 10H = Outside Segment 
for I/O Validation 


/O Map /0 Map 
Base Addres FFFFH Base Addres FFFFH 


—3> FFFFH + 10H = FH 
for I/O Validation 


OH OH 
/O access at port 10H checks /O access at port 10H checks 
bitmap at I/O map base address bitmap at I/O address FFFFH + 10H, 
FFFFH + 10H = offset 10H. which exceeds segment limit. 
Offset FH from beginning of Wrap around does not occur, 
TSS segment results because general-protection exception (#GP) 
wraparound occurs. occurs. 


Figure 21-1. I/O Map Base Address Differences 


21.29 CACHE MANAGEMENT 


The P6 family processors include two levels of internal caches: Li (level 1) and L2 (level 2). The L1 cache is divided 
into an instruction cache and a data cache; the L2 cache is a general-purpose cache. See Section 11.1, “Internal 
Caches, TLBs, and Buffers,” for a description of these caches. (Note that although the Pentium II processor L2 
cache is physically located on a separate chip in the cassette, it is considered an internal cache.) 


The Pentium processor includes separate level 1 instruction and data caches. The data cache supports a writeback 
(or alternatively write-through, on a line by line basis) policy for memory updates. 


The Intel486 processor includes a single level 1 cache for both instructions and data. 


The meaning of the CD and NW flags in control register CRO have been redefined for the P6 family and Pentium 
processors. For these processors, the recommended value (O0OB) enables writeback for the data cache of the 
Pentium processor and for the Li data cache and L2 cache of the P6 family processors. In the Intel486 processor, 
setting these flags to (00B) enables write-through for the cache. 


External system hardware can force the Pentium processor to disable caching or to use the write-through cache 
policy should that be required. In the P6 family processors, the MTRRs can be used to override the CD and NW flags 
(see Table 11-6). 


The P6 family and Pentium processors support page-level cache management in the same manner as the Intel486 
processor by using the PCD and PWT flags in control register CR3, the page-directory entries, and the page-table 
entries. The Intel486 processor, however, is not affected by the state of the PWT flag since the internal cache of the 
Intel486 processor is a write-through cache. 


21.29.1 Self-Modifying Code with Cache Enabled 


On the Intel486 processor, a write to an instruction in the cache will modify it in both the cache and memory. If the 
instruction was prefetched before the write, however, the old version of the instruction could be the one executed. 
To prevent this problem, it is necessary to flush the instruction prefetch unit of the Intel486 processor by coding a 
jump instruction immediately after any write that modifies an instruction. The P6 family and Pentium processors, 
however, check whether a write may modify an instruction that has been prefetched for execution. This check is 
based on the linear address of the instruction. If the linear address of an instruction is found to be present in the 
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prefetch queue, the P6 family and Pentium processors flush the prefetch queue, eliminating the need to code a 
jump instruction after any writes that modify an instruction. 


Because the linear address of the write is checked against the linear address of the instructions that have been 
prefetched, special care must be taken for self-modifying code to work correctly when the physical addresses of the 
instruction and the written data are the same, but the linear addresses differ. In such cases, it is necessary to 
execute a serializing operation to flush the prefetch queue after the write and before executing the modified 
instruction. See Section 8.3, “Serializing Instructions,” for more information on serializing instructions. 


NOTE 


The check on linear addresses described above is not in practice a concern for compatibility. Appli- 
cations that include self-modifying code use the same linear address for modifying and fetching the 
instruction. System software, such as a debugger, that might possibly modify an instruction using 
a different linear address than that used to fetch the instruction must execute a serializing 
operation, such as IRET, before the modified instruction is executed. 


21.29.2 Disabling the L3 Cache 


A unified third-level (L3) cache in processors based on Intel NetBurst microarchitecture (see Section 11.1, 
“Internal Caches, TLBs, and Buffers”) provides the third-level cache disable flag, bit 6 of the IA32_MISC_ENABLE 
MSR. The third-level cache disable flag allows the L3 cache to be disabled and enabled, independently of the L1 and 
L2 caches (see Section 11.5.4, “Disabling and Enabling the L3 Cache”). The third-level cache disable flag applies 
only to processors based on Intel NetBurst microarchitecture. Processors with L3 and based on other microarchi- 
tectures do not support the third-level cache disable flag. 


21.30 PAGING 


This section identifies enhancements made to the paging mechanism and implementation differences in the paging 
mechanism for various IA-32 processors. 


21.30.1 Large Pages 


The Pentium processor extended the memory management/paging facilities of the IA-32 to allow large (4 MBytes) 
pages sizes (see Section 4.3, “32-Bit Paging”). The first P6 family processor (the Pentium Pro processor) added a 
2 MByte page size to the IA-32 in conjunction with the physical address extension (PAE) feature (see Section 4.4, 
“PAE Paging”). 


The availability of large pages with 32-bit paging on any IA-32 processor can be determined via feature bit 3 (PSE) 
of register EDX after the CPUID instruction has been execution with an argument of 1. (Large pages are always 
available with PAE paging and 4-level paging.) Intel processors that do not support the CPUID instruction support 
only 32-bit paging and do not support page size enhancements. (See “CPUID—CPU Identification” in Chapter 3, 
“Instruction Set Reference, A-L,” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
2A for more information on the CPUID instruction. ) 


21.30.2 PCD and PWT Flags 


The PCD and PWT flags were introduced to the IA-32 in the Intel486 processor to control the caching of pages: 
¢ PCD (page-level cache disable) flag—Controls caching on a page-by-page basis. 


¢ PWT (page-level write-through) flag—Controls the write-through/writeback caching policy on a page-by-page 
basis. Since the internal cache of the Intel486 processor is a write-through cache, it is not affected by the state 
of the PWT flag. 
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21.30.3 Enabling and Disabling Paging 


Paging is enabled and disabled by loading a value into control register CRO that modifies the PG flag. For backward 
and forward compatibility with all IA-32 processors, Intel recommends that the following operations be performed 
when enabling or disabling paging: 


1. Execute a MOV CRO, REG instruction to either set (enable paging) or clear (disable paging) the PG flag. 
2. Execute a near JMP instruction. 


The sequence bounded by the MOV and JMP instructions should be identity mapped (that is, the instructions should 
reside on a page whose linear and physical addresses are identical). 


For the P6 family processors, the MOV CRO, REG instruction is serializing, so the jump operation is not required. 
However, for backwards compatibility, the JMP instruction should still be included. 


21.31 STACK OPERATIONS AND SUPERVISOR SOFTWARE 


This section identifies the differences in the stack mechanism for the various IA-32 processors. 


21.31.1 Selector Pushes and Pops 


When pushing a segment selector onto the stack, the Pentium 4, Intel Xeon, P6 family, and Intel486 processors 
decrement the ESP register by the operand size and then write 2 bytes. If the operand size is 32-bits, the upper two 
bytes of the write are not modified. The Pentium processor decrements the ESP register by the operand size and 
determines the size of the write by the operand size. If the operand size is 32-bits, the upper two bytes are written 
as Os. 


When popping a segment selector from the stack, the Pentium 4, Intel Xeon, P6 family, and Intel486 processors 
read 2 bytes and increment the ESP register by the operand size of the instruction. The Pentium processor deter- 
mines the size of the read from the operand size and increments the ESP register by the operand size. 


It is possible to align a 32-bit selector push or pop such that the operation generates an exception on a Pentium 
processor and not on an Pentium 4, Intel Xeon, P6 family, or Intel486 processor. This could occur if the third and/or 
fourth byte of the operation lies beyond the limit of the segment or if the third and/or fourth byte of the operation 
is locate on a non-present or inaccessible page. 


For a POP-to-memory instruction that meets the following conditions: 
¢ The stack segment size is 16-bit. 
e Any 32-bit addressing form with the SIB byte specifying ESP as the base register. 


¢ The initial stack pointer is FFFCH (32-bit operand) or FFFEH (16-bit operand) and will wrap around to OH as a 
result of the POP operation. 


The result of the memory write is implementation-specific. For example, in P6 family processors, the result of the 
memory write is SS:0H plus any scaled index and displacement. In Pentium processors, the result of the memory 
write may be either a stack fault (real mode or protected mode with stack segment size of 64 KByte), or write to 

SS:10000H plus any scaled index and displacement (protected mode and stack segment size exceeds 64 KByte). 


21.31.2 Error Code Pushes 


The Intel486 processor implements the error code pushed on the stack as a 16-bit value. When pushed onto a 32- 
bit stack, the Intel486 processor only pushes 2 bytes and updates ESP by 4. The P6 family and Pentium processors’ 
error code is a full 32 bits with the upper 16 bits set to zero. The P6 family and Pentium processors, therefore, push 
4 bytes and update ESP by 4. Any code that relies on the state of the upper 16 bits may produce inconsistent 
results. 
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21.31.3 Fault Handling Effects on the Stack 


During the handling of certain instructions, such as CALL and PUSHA, faults may occur in different sequences for 
the different processors. For example, during far calls, the Intel486 processor pushes the old CS and EIP before a 
possible branch fault is resolved. A branch fault is a fault from a branch instruction occurring from a segment limit 
or access rights violation. If a branch fault is taken, the Intel486 and P6 family processors will have corrupted 
memory below the stack pointer. However, the ESP register is backed up to make the instruction restartable. The 
P6 family processors issue the branch before the pushes. Therefore, if a branch fault does occur, these processors 
do not corrupt memory below the stack pointer. This implementation difference, however, does not constitute a 
compatibility problem, as only values at or above the stack pointer are considered to be valid. Other operations 
that encounter faults may also corrupt memory below the stack pointer and this behavior may vary on different 
implementations. 


21.31.4 Interlevel RET/IRET From a 16-Bit Interrupt or Call Gate 


If a call or interrupt is made from a 32-bit stack environment through a 16-bit gate, only 16 bits of the old ESP can 
be pushed onto the stack. On the subsequent RET/IRET, the 16-bit ESP is popped but the full 32-bit ESP is updated 
since control is being resumed in a 32-bit stack environment. The Intel486 processor writes the SS selector into the 
upper 16 bits of ESP. The P6 family and Pentium processors write zeros into the upper 16 bits. 


21.32 MIXING 16- AND 32-BIT SEGMENTS 


The features of the 16-bit Intel 286 processor are an object-code compatible subset of those of the 32-bit IA-32 
processors. The D (default operation size) flag in segment descriptors indicates whether the processor treats a 
code or data segment as a 16-bit or 32-bit segment; the B (default stack size) flag in segment descriptors indicates 
whether the processor treats a stack segment as a 16-bit or 32-bit segment. 


The segment descriptors used by the Intel 286 processor are supported by the 32-bit IA-32 processors if the Intel- 
reserved word (highest word) of the descriptor is clear. On the 32-bit IA-32 processors, this word includes the 
upper bits of the base address and the segment limit. 


The segment descriptors for data segments, code segments, local descriptor tables (there are no descriptors for 
global descriptor tables), and task gates are the same for the 16- and 32-bit processors. Other 16-bit descriptors 
(TSS segment, call gate, interrupt gate, and trap gate) are supported by the 32-bit processors. 


The 32-bit processors also have descriptors for TSS segments, call gates, interrupt gates, and trap gates that 
support the 32-bit architecture. Both kinds of descriptors can be used in the same system. 


For those segment descriptors common to both 16- and 32-bit processors, clear bits in the reserved word cause the 
32-bit processors to interpret these descriptors exactly as an Intel 286 processor does, that is: 


¢ Base Address — The upper 8 bits of the 32-bit base address are clear, which limits base addresses to 24 bits. 
¢ Limit — The upper 4 bits of the limit field are clear, restricting the value of the limit field to 64 KBytes. 


°® Granularity bit — The G (granularity) flag is clear, indicating the value of the 16-bit limit is interpreted in units 
of 1 byte. 


° Big bit — In a data-segment descriptor, the B flag is clear in the segment descriptor used by the 32-bit 
processors, indicating the segment is no larger than 64 KBytes. 


° Default bit — In a code-segment descriptor, the D flag is clear, indicating 16-bit addressing and operands are 
the default. In a stack-segment descriptor, the D flag is clear, indicating use of the SP register (instead of the 
ESP register) and a 64-KByte maximum segment limit. 


For information on mixing 16- and 32-bit code in applications, see Chapter 20, “Mixing 16-Bit and 32-Bit Code.” 


21.33 SEGMENT AND ADDRESS WRAPAROUND 


This section discusses differences in segment and address wraparound between the P6 family, Pentium, Intel486, 
Intel386, Intel 286, and 8086 processors. 
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21.33.1 Segment Wraparound 


On the 8086 processor, an attempt to access a memory operand that crosses offset 65,535 or OFFFFH or offset 0 
(for example, moving a word to offset 65,535 or pushing a word when the stack pointer is set to 1) causes the 
offset to wrap around modulo 65,536 or 010000H. With the Intel 286 processor, any base and offset combination 
that addresses beyond 16 MBytes wraps around to the 1 MByte of the address space. The P6 family, Pentium, 
Intel486, and Intel386 processors in real-address mode generate an exception in these cases: 


¢ A general-protection exception (#GP) if the segment is a data segment (that is, if the CS, DS, ES, FS, or GS 
register is being used to address the segment). 


e A stack-fault exception (#SS) if the segment is a stack segment (that is, if the SS register is being used). 


An exception to this behavior occurs when a stack access is data aligned, and the stack pointer is pointing to the 
last aligned piece of data that size at the top of the stack (ESP is FFFFFFFCH). When this data is popped, no 
segment limit violation occurs and the stack pointer will wrap around to 0. 


The address space of the P6 family, Pentium, and Intel486 processors may wraparound at 1 MByte in real-address 
mode. An external A20M# pin forces wraparound if enabled. On Intel 8086 processors, it is possible to specify 
addresses greater than 1 MByte. For example, with a selector value FFFFH and an offset of FFFFH, the effective 
address would be 10FFEFH (1 MByte plus 65519 bytes). The 8086 processor, which can form addresses up to 20 
bits long, truncates the uppermost bit, which “wraps” this address to FFEFH. However, the P6 family, Pentium, and 
Intel486 processors do not truncate this bit if AZOM# is not enabled. 


If a stack operation wraps around the address limit, shutdown occurs. (The 8086 processor does not have a shut- 
down mode or a limit.) 


The behavior when executing near the limit of a 4-GByte selector (limit = FFFFFFFFH) is different between the 
Pentium Pro and the Pentium 4 family of processors. On the Pentium Pro, instructions which cross the limit -- for 
example, a two byte instruction such as INC EAX that is encoded as FFH COH starting exactly at the limit faults for 
a segment violation (a one byte instruction at FFFFFFFFH does not cause an exception). Using the Pentium 4 micro- 
processor family, neither of these situations causes a fault. 


Segment wraparound and the functionality of A20M# is used primarily by older operating systems and not used by 
modern operating systems. On newer Intel 64 processors, AZOM# may be absent. 


21.34 STORE BUFFERS AND MEMORY ORDERING 


The Pentium 4, Intel Xeon, and P6 family processors provide a store buffer for temporary storage of writes (stores) 
to memory (see Section 11.10, “Store Buffer”). Writes stored in the store buffer(s) are always written to memory 
in program order, with the exception of “fast string” store operations (see Section 8.2.4, “Fast-String Operation and 
Out-of-Order Stores”). 


The Pentium processor has two store buffers, one corresponding to each of the pipelines. Writes in these buffers 
are always written to memory in the order they were generated by the processor core. 


It should be noted that only memory writes are buffered and I/O writes are not. The Pentium 4, Intel Xeon, P6 
family, Pentium, and Intel486 processors do not synchronize the completion of memory writes on the bus and 
instruction execution after a write. An I/O, locked, or serializing instruction needs to be executed to synchronize 
writes with the next instruction (see Section 8.3, “Serializing Instructions”). 


The Pentium 4, Intel Xeon, and P6 family processors use processor ordering to maintain consistency in the order 
that data is read (loaded) and written (stored) in a program and the order the processor actually carries out the 
reads and writes. With this type of ordering, reads can be carried out speculatively and in any order, reads can pass 
buffered writes, and writes to memory are always carried out in program order. (See Section 8.2, “Memory 
Ordering,” for more information about processor ordering.) The Pentium III processor introduced a new instruction 
to serialize writes and make them globally visible. Memory ordering issues can arise between a producer anda 
consumer of data. The SFENCE instruction provides a performance-efficient way of ensuring ordering between 
routines that produce weakly-ordered results and routines that consume this data. 


No re-ordering of reads occurs on the Pentium processor, except under the condition noted in Section 8.2.1, 
“Memory Ordering in the Intel® Pentium® and Intel486" Processors,” and in the following paragraph describing the 
Intel486 processor. 
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Specifically, the store buffers are flushed before the IN instruction is executed. No reads (as a result of cache miss) 
are reordered around previously generated writes sitting in the store buffers. The implication of this is that the 
store buffers will be flushed or emptied before a subsequent bus cycle is run on the external bus. 


On both the Intel486 and Pentium processors, under certain conditions, a memory read will go onto the external 
bus before the pending memory writes in the buffer even though the writes occurred earlier in the program execu- 
tion. A memory read will only be reordered in front of all writes pending in the buffers if all writes pending in the 
buffers are cache hits and the read is a cache miss. Under these conditions, the Intel486 and Pentium processors 
will not read from an external memory location that needs to be updated by one of the pending writes. 


During a locked bus cycle, the Intel486 processor will always access external memory, it will never look for the 
location in the on-chip cache. All data pending in the Intel486 processor's store buffers will be written to memory 
before a locked cycle is allowed to proceed to the external bus. Thus, the locked bus cycle can be used for elimi- 
nating the possibility of reordering read cycles on the Intel486 processor. The Pentium processor does check its 
cache on a read-modify-write access and, if the cache line has been modified, writes the contents back to memory 
before locking the bus. The P6 family processors write to their cache on a read-modify-write operation (if the 
access does not split across a cache line) and does not write back to system memory. If the access does split across 
a cache line, it locks the bus and accesses system memory. 


I/O reads are never reordered in front of buffered memory writes on an IA-32 processor. This ensures an update of 
all memory locations before reading the status from an I/O device. 


21.35 BUS LOCKING 


The Intel 286 processor performs the bus locking differently than the Intel P6 family, Pentium, Intel486, and 
Intel386 processors. Programs that use forms of memory locking specific to the Intel 286 processor may not run 
properly when run on later processors. 


A locked instruction is guaranteed to lock only the area of memory defined by the destination operand, but may 
lock a larger memory area. For example, typical 8086 and Intel 286 configurations lock the entire physical memory 
space. Programmers should not depend on this. 


On the Intel 286 processor, the LOCK prefix is sensitive to IOPL. If the CPL is greater than the IOPL, a general- 
protection exception (#GP) is generated. On the Intel386 DX, Intel486, and Pentium, and P6 family processors, no 
check against IOPL is performed. 


The Pentium processor automatically asserts the LOCK# signal when acknowledging external interrupts. After 
signaling an interrupt request, an external interrupt controller may use the data bus to send the interrupt vector to 
the processor. After receiving the interrupt request signal, the processor asserts LOCK# to ensure that no other 
data appears on the data bus until the interrupt vector is received. This bus locking does not occur on the P6 family 
processors. 


21.36 BUS HOLD 


Unlike the 8086 and Intel 286 processors, but like the Intel386 and Intel486 processors, the P6 family and Pentium 
processors respond to requests for control of the bus from other potential bus masters, such as DMA controllers, 
between transfers of parts of an unaligned operand, such as two words which form a doubleword. Unlike the 
Intel386 processor, the P6 family, Pentium and Intel486 processors respond to bus hold during reset initialization. 


21.37  MODEL-SPECIFIC EXTENSIONS TO THE IA-32 


Certain extensions to the IA-32 are specific to a processor or family of IA-32 processors and may not be imple- 
mented or implemented in the same way in future processors. The following sections describe these model-specific 
extensions. The CPUID instruction indicates the availability of some of the model-specific features. 
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21.37.1 Model-Specific Registers 


The Pentium processor introduced a set of model-specific registers (MSRs) for use in controlling hardware functions 
and performance monitoring. To access these MSRs, two new instructions were added to the IA-32 architecture: 
read MSR (RDMSR) and write MSR (WRMSR). The MSRs in the Pentium processor are not guaranteed to be dupli- 
cated or provided in the next generation IA-32 processors. 


The P6 family processors greatly increased the number of MSRs available to software. See Chapter 2, “Model- 
Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4 for a 
complete list of the available MSRs. The new registers control the debug extensions, the performance counters, the 
machine-check exception capability, the machine-check architecture, and the MTRRs. These registers are acces- 
sible using the RDMSR and WRMSR instructions. Specific information on some of these new MSRs is provided in the 
following sections. As with the Pentium processor MSR, the P6 family processor MSRs are not guaranteed to be 
duplicated or provided in the next generation IA-32 processors. 


21.37.2 RDMSR and WRMSR Instructions 


The RDMSR (read model-specific register) and WRMSR (write model-specific register) instructions recognize a 
much larger number of model-specific registers in the P6 family processors. (See “RDMSR—Read from Model 
Specific Register” and “WRMSR—Write to Model Specific Register” in the Intel® 64 and IA-32 Architectures Soft- 
ware Developer’s Manual, Volumes 2A, 2B, 2C & 2D for more information.) 


21.37.3. Memory Type Range Registers 


Memory type range registers (MTRRs) are a new feature introduced into the IA-32 in the Pentium Pro processor. 
MTRRs allow the processor to optimize memory operations for different types of memory, such as RAM, ROM, frame 
buffer memory, and memory-mapped I/O. 


MTRRs are MSRs that contain an internal map of how physical address ranges are mapped to various types of 
memory. The processor uses this internal memory map to determine the cacheability of various physical memory 
locations and the optimal method of accessing memory locations. For example, if a memory location is specified in 
an MTRR as write-through memory, the processor handles accesses to this location as follows. It reads data from 
that location in lines and caches the read data or maps all writes to that location to the bus and updates the cache 
to maintain cache coherency. In mapping the physical address space with MTRRs, the processor recognizes five 
types of memory: uncacheable (UC), uncacheable, speculatable, write-combining (WC), write-through (WT), 
write-protected (WP), and writeback (WB). 


Earlier IA-32 processors (such as the Intel486 and Pentium processors) used the KEN# (cache enable) pin and 
external logic to maintain an external memory map and signal cacheable accesses to the processor. The MTRR 
mechanism simplifies hardware designs by eliminating the KEN# pin and the external logic required to drive it. 


See Chapter 9, “Processor Management and Initialization,” and Chapter 2, “Model-Specific Registers (MSRs)” in the 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4 for more information on the MTRRs. 


21.37.4 Machine-Check Exception and Architecture 


The Pentium processor introduced a new exception called the machine-check exception (#MC, interrupt 18). This 
exception is used to detect hardware-related errors, such as a parity error on a read cycle. 


The P6 family processors extend the types of errors that can be detected and that generate a machine-check 
exception. It also provides a new machine-check architecture for recording information about a machine-check 
error and provides extended recovery capability. 


The machine-check architecture provides several banks of reporting registers for recording machine-check errors. 
Each bank of registers is associated with a specific hardware unit in the processor. The primary focus of the 
machine checks is on bus and interconnect operations; however, checks are also made of translation lookaside 
buffer (TLB) and cache operations. 
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The machine-check architecture can correct some errors automatically and allow for reliable restart of instruction 
execution. It also collects sufficient information for software to use in correcting other machine errors not corrected 
by hardware. 


See Chapter 15, “Machine-Check Architecture,” for more information on the machine-check exception and the 
machine-check architecture. 


21.37.5 Performance-Monitoring Counters 


The P6 family and Pentium processors provide two performance-monitoring counters for use in monitoring internal 
hardware operations. The number of performance monitoring counters and associated programming interfaces 
may be implementation specific for Pentium 4 processors, Pentium M processors. Later processors may have 
implemented these as part of an architectural performance monitoring feature. The architectural and non-architec- 
tural performance monitoring interfaces for different processor families are described in Chapter 18, “Performance 
Monitoring,”. https://perfmon-events.intel.com/ lists all the events that can be counted for architectural perfor- 
mance monitoring events and non-architectural events. The counters are set up, started, and stopped using two 
MSRs and the RDMSR and WRMSR instructions. For the P6 family processors, the current count for a particular 
counter can be read using the new RDPMC instruction. 


The performance-monitoring counters are useful for debugging programs, optimizing code, diagnosing system fail- 
ures, or refining hardware designs. See Chapter 18, “Performance Monitoring,” for more information on these 
counters. 


21.38 TWO WAYS TO RUN INTEL 286 PROCESSOR TASKS 


When porting 16-bit programs to run on 32-bit IA-32 processors, there are two approaches to consider: 


° Porting an entire 16-bit software system to a 32-bit processor, complete with the old operating system, loader, 
and system builder. Here, all tasks will have 16-bit TSSs. The 32-bit processor is being used as if it were a faster 
version of the 16-bit processor. 


¢ Porting selected 16-bit applications to run in a 32-bit processor environment with a 32-bit operating system, 
loader, and system builder. Here, the TSSs used to represent 286 tasks should be changed to 32-bit TSSs. It is 
possible to mix 16 and 32-bit TSSs, but the benefits are small and the problems are great. All tasks in a 32-bit 
software system should have 32-bit TSSs. It is not necessary to change the 16-bit object modules themselves; 
TSSs are usually constructed by the operating system, by the loader, or by the system builder. See Chapter 20, 
“Mixing 16-Bit and 32-Bit Code,” for more detailed information about mixing 16-bit and 32-bit code. 


Because the 32-bit processors use the contents of the reserved word of 16-bit segment descriptors, 16-bit 
programs that place values in this word may not run correctly on the 32-bit processors. 


21.39 INITIAL STATE OF PENTIUM, PENTIUM PRO AND PENTIUM 4 PROCESSORS 


Table 21-10 shows the state of the flags and other registers following power-up for the Pentium, Pentium Pro and 
Pentium 4 processors. The state of control register CRO is 6(0000010H (see Figure 9-1 “Contents of CRO Register 

after Reset” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). This places the 

processor in real-address mode with paging disabled. 


Table 21-10. Processor State Following Power-up/Reset/INIT for Pentium, Pentium Pro and Pentium 4 Processors 


Register Pentium 4 Processor Pentium Pro Processor Pentium Processor 
EFLAGS! 00000002H 00000002H 00000002H 
EIP OOOOFFFOH OOOOFFFOH OOOOFFFOH 
CRO 60000010H? 60000010H* 60000010H? 
CR2, CR3, CR4 OO000000H O00000000H O0000000H 
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Table 21-10. Processor State Following Power-up/Reset/INIT for Pentium, Pentium Pro and Pentium 4 Processors 


Register 


Pentium 4 Processor 


Pentium Pro Processor 


Pentium Processor 


cS 


Selector = FOOOH 

Base = FFFFOOOOH 

Limit = FFFFH 

AR = Present, R/W, Accessed 


Selector = FOOOH 

Base = FFFFOOOOH 

Limit = FFFFH 

AR = Present, R/W, Accessed 


Selector = FOOOH 

Base = FFFFOOOOH 

Limit = FFFFH 

AR = Present, R/W, Accessed 


SS, DS, ES, FS, GS 


Selector = OOOOH 
Base = QOOO00000H 


Selector = OOOOH 
Base = OOOO0000H 


Selector = OOOOH 
Base = OOOO00000H 


Limit = FFFFH Limit = FFFFH Limit = FFFFH 

AR = Present, R/W, Accessed AR = Present, R/W, Accessed AR = Present, R/W, Accessed 
EDX OOOO0FxxH 000n06xxH? 000005xxH 
EAX ot o* o* 
EBX, ECX, ESI, EDI, EBP, | COOOO000H O0000000H OO000000H 


ESP 


STO through ST7° 


Pwr up or Reset: +0.0 
FINIT/FNINIT: Unchanged 


Pwr up or Reset: +0.0 
FINIT/FNINIT: Unchanged 


Pwr up or Reset: +0.0 
FINIT/FNINIT: Unchanged 


x87 FPU Control 
Word? 


x87 FPU Status Word? 


Pwr up or Reset: 0040H 
FINIT/FNINIT: 037FH 


Pwr up or Reset: OOOOH 
FINIT/FNINIT: OOOOH 


Pwr up or Reset: 0040H 
FINIT/FNINIT: 037FH 


Pwr up or Reset: OOOOH 
FINIT/FNINIT: OOOOH 


Pwr up or Reset: 0040H 
FINIT/FNINIT: 037FH 


Pwr up or Reset: OOOOH 
FINIT/FNINIT: OOOOH 


x87 FPU Tag Word? 


x87 FPU Data 
Operand and CS Seg. 
Selectors? 


Pwr up or Reset: 5555H 
FINIT/FNINIT: FFFFH 


Pwr up or Reset: OOOOH 
FINIT/FNINIT: OOOOH 


Pwr up or Reset: 5555H 
FINIT/FNINIT: FFFFH 


Pwr up or Reset: OOOOH 
FINIT/FNINIT: OOOOH 


Pwr up or Reset: 5555H 
FINIT/FNINIT: FFFFH 


Pwr up or Reset: OOOOH 
FINIT/FNINIT: OOOOH 


x87 FPU Data 
Operand and Inst. 
Pointers? 


Pwr up or Reset: 
O0000000H 
FINIT/FNINIT: OOOOOOOOH 


Pwr up or Reset: 
QO000000H 
FINIT/FNINIT: OOOOOOOOH 


Pwr up or Reset: 
OO000000H 
FINIT/FNINIT: OOOOOOOOH 


MMO through MM7° 


Pwr up or Reset: 


Pentium II and Pentium Ill 


Pentium with MMX Technology 


OOOO0000000000000H Processors Only— Only— 
INIT or FINIT/FNINIT: Pwr up or Reset: Pwr up or Reset: 
Unchanged 0000000000000000H 0000000000000000H 
INIT or FINIT/FNINIT: INIT or FINIT/FNINIT: 
Unchanged Unchanged 
XMMO through XMM7_ | Pwr up or Reset: OH If CPUID.O1H:SSE is 1 — NA 
INIT: Unchanged Pwr up or Reset: OH 
INIT: Unchanged 
MXCSR Pwr up or Reset: 1F80H Pentium Ill processor only- NA 
INIT: Unchanged Pwr up or Reset: 1F80H 
INIT: Unchanged 
GDTR, IDTR Base = OQOO000000H Base = QOOOO000H Base = QOO00000H 


LDTR, Task Register 


Limit = FFFFH 
AR = Present, R/W 


Selector = OOOOH 
Base = OOOO00000H 


Limit = FFFFH 
AR = Present, R/W 


Selector = OOOOH 
Base = QOOO00000H 


Limit = FFFFH 
AR = Present, R/W 


Selector = OOOOH 
Base = OOO00000H 


Limit = FFFFH Limit = FFFFH Limit = FFFFH 

AR = Present, R/W AR = Present, R/W AR = Present, R/W 
DRO, DR1, DR2, DR3 OOOO0000H OOOO0000H OOO000000H 
DR6 FFFFOFFOH FFFFOFFOH FFFFOFFOH 
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Table 21-10. Processor State Following Power-up/Reset/INIT for Pentium, Pentium Pro and Pentium 4 Processors 


Register 


Pentium 4 Processor 


Pentium Pro Processor 


Pentium Processor 


DR7 


00000400H 


00000400H 


00000400H 


Time-Stamp Counter 


Power up or Reset: OH 
INIT: Unchanged 


Power up or Reset: OH 
INIT: Unchanged 


Power up or Reset: OH 
INIT: Unchanged 


Perf. Counters and 
Event Select 


Power up or Reset: OH 
INIT: Unchanged 


Power up or Reset: OH 
INIT: Unchanged 


Power up or Reset: OH 
INIT: Unchanged 


All Other MSRs 


Pwr up or Reset: 


Pwr up or Reset: 


Pwr up or Reset: 


Undefined Undefined Undefined 
INIT: Unchanged INIT: Unchanged INIT: Unchanged 
Data and Code Cache, | Invalid® Invalid® Invalid® 


TLBs 


Fixed MTRRs Pwr up or Reset: Disabled Pwr up or Reset: Disabled Not Implemented 
INIT: Unchanged INIT: Unchanged 
Variable MTRRs Pwr up or Reset: Disabled Pwr up or Reset: Disabled Not Implemented 


INIT: Unchanged 


INIT: Unchanged 


Machine-Check 
Architecture 


Pwr up or Reset: 
Undefined 
INIT: Unchanged 


Pwr up or Reset: 
Undefined 
INIT: Unchanged 


Not Implemented 


APIC Pwr up or Reset: Enabled Pwr up or Reset: Enabled Pwr up or Reset: Enabled 
INIT: Unchanged INIT: Unchanged INIT: Unchanged 

R8-R15” QOO0000000000000H QOO00000000000000H NA. 

XMM8-XMM157 Pwr up or Reset: OH Pwr up or Reset: OH N.A. 


INIT: Unchanged 


INIT: Unchanged 


NOTES: 


1. The 10 most-significant bits of the EFLAGS register are undefined following a reset. Software should not depend on the states of 


any of these bits. 


2. The CD and NW flags are unchanged, bit 4 is set to 1, all other bits are cleared. 
3. Where “n” is the Extended Model Value for the respective processor. 

4. If Built-In Self-Test (BIST) is invoked on power up or reset, EAX is O only if all tests passed. (BIST cannot be invoked during an INIT.) 
5. The state of the x87 FPU and MMX registers is not changed by the execution of an INIT. 
6. Internal caches are invalid after power-up and RESET, but left unchanged with an INIT. 
7. \f the processor supports IA-32e mode. 
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CHAPTER 22 
INTRODUCTION TO VIRTUAL MACHINE EXTENSIONS 


22.1 OVERVIEW 


This chapter describes the basics of virtual machine architecture and an overview of the virtual-machine extensions 
(VMX) that support virtualization of processor hardware for multiple software environments. 


Information about VMX instructions is provided in Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 2B. Other aspects of VMX and system programming considerations are described in chapters of Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 3B. 


22.2 VIRTUAL MACHINE ARCHITECTURE 


Virtual-machine extensions define processor-level support for virtual machines on IA-32 processors. Two principal 
classes of software are supported: 


° = Virtual-machine monitors (VMM) — A VMM acts as a host and has full control of the processor(s) and other 
platform hardware. A VMM presents guest software (see next paragraph) with an abstraction of a virtual 
processor and allows it to execute directly on a logical processor. A VMM is able to retain selective control of 
processor resources, physical memory, interrupt management, and I/O. 


¢ Guest software — Each virtual machine (VM) is a guest software environment that supports a stack consisting 
of operating system (OS) and application software. Each operates independently of other virtual machines and 
uses on the same interface to processor(s), memory, storage, graphics, and I/O provided by a physical 
platform. The software stack acts as if it were running on a platform with no VMM. Software executing ina 
virtual machine must operate with reduced privilege so that the VMM can retain control of platform resources. 


22.3. INTRODUCTION TO VMX OPERATION 


Processor support for virtualization is provided by a form of processor operation called VMX operation. There are 
two kinds of VMX operation: VMX root operation and VMX non-root operation. In general, a VMM will run in VMX 
root operation and guest software will run in VMX non-root operation. Transitions between VMX root operation and 
VMX non-root operation are called VMX transitions. There are two kinds of VMX transitions. Transitions into VMX 
non-root operation are called VM entries. Transitions from VMX non-root operation to VMX root operation are called 
VM exits. 


Processor behavior in VMX root operation is very much as it is outside VMX operation. The principal differences are 
that a set of new instructions (the VMX instructions) is available and that the values that can be loaded into certain 
control registers are limited (see Section 22.8). 


Processor behavior in VMX non-root operation is restricted and modified to facilitate virtualization. Instead of their 
ordinary operation, certain instructions (including the new VMCALL instruction) and events cause VM exits to the 
VMM. Because these VM exits replace ordinary behavior, the functionality of software in VMX non-root operation is 
limited. It is this limitation that allows the VMM to retain control of processor resources. 


There is no software-visible bit whose setting indicates whether a logical processor is in VMX non-root operation. 
This fact may allow a VMM to prevent guest software from determining that it is running in a virtual machine. 


Because VMX operation places restrictions even on software running with current privilege level (CPL) 0, guest 
software can run at the privilege level for which it was originally designed. This capability may simplify the devel- 
opment of a VMM. 


Vol. 3C 22-1 


INTRODUCTION TO VIRTUAL MACHINE EXTENSIONS 


22.4 LIFE CYCLE OF VMM SOFTWARE 


Figure 22-1 illustrates the life cycle of a VMM and its guest software as well as the interactions between them. The 
following items summarize that life cycle: 


° Software enters VMX operation by executing a VMXON instruction. 


¢ Using VM entries, a VMM can then enter guests into virtual machines (one at a time). The VMM effects a 
VM entry using instructions VMLAUNCH and VMRESUME; it regains control using VM exits. 

¢ VM exits transfer control to an entry point specified by the VMM. The VMM can take action appropriate to the 
cause of the VM exit and can then return to the virtual machine using a VM entry. 


° Eventually, the VMM may decide to shut itself down and leave VMX operation. It does so by executing the 
VMXOFF instruction. 


Guest 0 Guest 1 


VM Exit VM Exit 


VMXON VM Monitor VMXOFF 


Figure 22-1. Interaction of a Virtual-Machine Monitor and Guests 


22.5 VIRTUAL-MACHINE CONTROL STRUCTURE 


VMX non-root operation and VMX transitions are controlled by a data structure called a virtual-machine control 
structure (VMCS). 


Access to the VMCS is managed through a component of processor state called the VMCS pointer (one per logical 
processor). The value of the VMCS pointer is the 64-bit address of the VMCS. The VMCS pointer is read and written 
using the instructions VMPTRST and VMPTRLD. The VMM configures a VMCS using the VMREAD, VMWRITE, and 
VMCLEAR instructions. 


AVMM could use a different VMCS for each virtual machine that it supports. For a virtual machine with multiple 
logical processors (virtual processors), the VMM could use a different VMCS for each virtual processor. 


22.6 DISCOVERING SUPPORT FOR VMX 


Before system software enters into VMX operation, it must discover the presence of VMX support in the processor. 
System software can determine whether a processor supports VMX operation using CPUID. If 
CPUID.1:ECX.VMX[bit 5] = 1, then VMX operation is supported. See Chapter 3, “Instruction Set Reference, A-L” of 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2A. 


The VMX architecture is designed to be extensible so that future processors in VMX operation can support addi- 
tional features not present in first-generation implementations of the VMX architecture. The availability of exten- 
sible VMX features is reported to software using a set of VMX capability MSRs (see Appendix A, “VMX Capability 
Reporting Facility”). 
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22.7. ENABLING AND ENTERING VMX OPERATION 


Before system software can enter VMX operation, it enables VMX by setting CR4.VMXE[bit 13] = 1. VMX operation 
is then entered by executing the VMXON instruction. VMXON causes an invalid-opcode exception (#UD) if executed 
with CR4.VMXE = 0. Once in VMX operation, it is not possible to clear CR4.VMXE (see Section 22.8). System soft- 
ware leaves VMX operation by executing the VMXOFF instruction. CR4.VMXE can be cleared outside of VMX opera- 
tion after executing of VMXOFF. 


VMXON is also controlled by the IA32_FEATURE_CONTROL MSR (MSR address 3AH). This MSR is cleared to zero 
when a logical processor is reset. The relevant bits of the MSR are: 


° Bit O is the lock bit. If this bit is clear, VMXON causes a general-protection exception. If the lock bit is set, 
WRMSR to this MSR causes a general-protection exception; the MSR cannot be modified until a power-up reset 
condition. System BIOS can use this bit to provide a setup option for BIOS to disable support for VMX. To 
enable VMX support in a platform, BIOS must set bit 1, bit 2, or both (see below), as well as the lock bit. 


¢ Bit 1 enables VMXON in SMX operation. If this bit is clear, execution of VMXON in SMX operation causes a 
general-protection exception. Attempts to set this bit on logical processors that do not support both VMX 
operation (see Section 22.6) and SMX operation (see Chapter 6, “Safer Mode Extensions Reference,” in Inte/® 
64 and IA-32 Architectures Software Developer’s Manual, Volume 2D) cause general-protection exceptions. 


¢ Bit 2 enables VMXON outside SMX operation. If this bit is clear, execution of VMXON outside SMX 
operation causes a general-protection exception. Attempts to set this bit on logical processors that do not 
support VMX operation (see Section 22.6) cause general-protection exceptions. 


NOTE 


A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last 
execution of GETSEC[SENTER]. A logical processor is outside SMX operation if GETSEC[SENTER] 
has not been executed or if GETSEC[SEXIT] was executed after the last execution of 
GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference,” in Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 2D. 


Before executing VMXON, software should allocate a naturally aligned 4-KByte region of memory that a logical 
processor may use to support VMxX operation.! This region is called the VMXON region. The address of the VMXON 
region (the VMXON pointer) is provided in an operand to VMXON. Section 23.11.5, “VMXON Region,” details how 
software should initialize and access the VMXON region. 


22.8 RESTRICTIONS ON VMX OPERATION 


VMX operation places restrictions on processor operation. These are detailed below: 


° In VMX operation, processors may fix certain bits in CRO and CR4 to specific values and not support other 
values. VMXON fails if any of these bits contains an unsupported value (see “VMXON—Enter VMX Operation” in 
Chapter 29). Any attempt to set one of these bits to an unsupported value while in VMX operation (including 
VMX root operation) using any of the CLTS, LMSW, or MOV CR instructions causes a general-protection 
exception. VM entry or VM exit cannot set any of these bits to an unsupported value. Software should consult 
the VMX capability MSRs IA32_VMX_CRO_FIXEDO and IA32_VMX_CRO_FIXED1 to determine how bits in CRO 
are fixed (see Appendix A.7). For CR4, software should consult the VMX capability MSRs 
IA32_VMX_CR4_FIXEDO and IA32_VMX_CR4_FIXED1 (see Appendix A.8). 


NOTES 


The first processors to support VMX operation require that the following bits be 1 in VMX operation: 
CRO.PE, CRO.NE, CRO.PG, and CR4.VMXE. The restrictions on CRO.PE and CRO.PG imply that VMX 


1. Future processors may require that a different amount of memory be reserved. If so, this fact is reported to software using the 
V/MX capability-reporting mechanism. 


Vol. 3C 22-3 


INTRODUCTION TO VIRTUAL MACHINE EXTENSIONS 


operation is supported only in paged protected mode (including IA-32e mode). Therefore, guest 
software cannot be run in unpaged protected mode or in real-address mode. 


Later processors support a VM-execution control called “unrestricted guest” (see Section 23.6.2). 
If this control is 1, CRO.PE and CRO.PG may be 0 in VMX non-root operation (even if the capability 
MSR IA32_VMX_CRO_FIXEDO reports otherwise). Such processors allow guest software to run in 
unpaged protected mode or in real-address mode. 


¢ VMXON fails if a logical processor is in A20M mode (see “VMXON—Enter VMX Operation” in Chapter 29). Once 
the processor is in VMX operation, A20M interrupts are blocked. Thus, it is impossible to be in A20M mode in 
VMX operation. 


¢ The INIT signal is blocked whenever a logical processor is in VMX root operation. It is not blocked in VMX non- 
root operation. Instead, INITs cause VM exits (see Section 24.2, “Other Causes of VM Exits”). 


* Intel® Processor Trace (Intel PT) can be used in VMX operation only if IA32_VMX_MISC[14] is read as 1 (see 
Appendix A.6). On processors that support Intel PT but which do not allow it to be used in VMX operation, 
execution of VMXON clears IA32_RTIT_CTL.TraceEn (see “VMXON—Enter VMX Operation” in Chapter 29); any 
attempt to write IA32_RTIT_CTL while in VMX operation (including VMX root operation) causes a general- 
protection exception. 


1. “Unrestricted guest” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VMX non-root operation functions as if the “unrestricted guest” VM-execution control were 0. See Section 23.6.2. 
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23.1 OVERVIEW 


A logical processor uses virtual-machine control data structures (VMCSs) while it is in VMX operation. These 
manage transitions into and out of VMX non-root operation (VM entries and VM exits) as well as processor behavior 
in VMX non-root operation. This structure is manipulated by the new instructions VMCLEAR, VMPTRLD, VMREAD, 
and VMWRITE. 


A VMM can use a different VMCS for each virtual machine that it supports. For a virtual machine with multiple 
logical processors (virtual processors), the VMM can use a different VMCS for each virtual processor. 


A logical processor associates a region in memory with each VMCS. This region is called the VMCS region.! Soft- 
ware references a specific VMCS using the 64-bit physical address of the region (a VMCS pointer). VMCS pointers 
must be aligned on a 4-KByte boundary (bits 11:0 must be zero). These pointers must not set bits beyond the 
processor's physical-address width.2/> 


A logical processor may maintain a number of VMCSs that are active. The processor may optimize VMX operation 
by maintaining the state of an active VMCS in memory, on the processor, or both. At any given time, at most one 
of the active VMCSs is the current VMCS. (This document frequently uses the term “the VMCS” to refer to the 
current VMCS.) The VMLAUNCH, VMREAD, VMRESUME, and VMWRITE instructions operate only on the current 
VMCS. 


The following items describe how a logical processor determines which VMCSs are active and which is current: 


¢ The memory operand of the VMPTRLD instruction is the address of a VMCS. After execution of the instruction, 
that VMCS is both active and current on the logical processor. Any other VMCS that had been active remains so, 
but no other VMCS is current. 


¢ The VMCS link pointer field in the current VMCS (see Section 23.4.2) is itself the address of a VMCS. If VM entry 
is performed successfully with the 1-setting of the “VMCS shadowing” VM-execution control, the VMCS 
referenced by the VMCS link pointer field becomes active on the logical processor. The identity of the current 
VMCS does not change. 


¢ The memory operand of the VMCLEAR instruction is also the address of a VMCS. After execution of the 
instruction, that VMCS is neither active nor current on the logical processor. If the VMCS had been current on 
the logical processor, the logical processor no longer has a current VMCS. 


The VMPTRST instruction stores the address of the logical processor’s current VMCS into a specified memory loca- 
tion (it stores the value FFFFFFFF_FFFFFFFFH if there is no current VMCS). 


The launch state of a VMCS determines which VM-entry instruction should be used with that VMCS: the 
VMLAUNCH instruction requires a VMCS whose launch state is “clear”; the VMRESUME instruction requires a VMCS 
whose launch state is “launched”. A logical processor maintains a VMCS’s launch state in the corresponding VMCS 
region. The following items describe how a logical processor manages the launch state of a VMCS: 


¢ If the launch state of the current VMCS is “clear”, successful execution of the VMLAUNCH instruction changes 
the launch state to “launched”. 


¢ The memory operand of the VMCLEAR instruction is the address of a VMCS. After execution of the instruction, 
the launch state of that VMCS is “clear”. 


¢ There are no other ways to modify the launch state of a VMCS (it cannot be modified using VMWRITE) and there 
is no direct way to discover it (it cannot be read using VMREAD). 


1. The amount of memory required for a VMCS region is at most 4 KBytes. The exact size is implementation specific and can be deter- 
mined by consulting the VMX capability MSR IA32_VMX_BASIC to determine the size of the VMCS region (see Appendix A.1). 


2. Software can determine a processor's physical-address width by executing CPUID with BOOO0008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


3. If IA32_VMX_BASIC[48] is read as 1, these pointers must not set any bits in the range 63:32; see Appendix A.1. 
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Figure 23-1 illustrates the different states of a VMCS. It uses “X” to refer to the VMCS and “Y” to refer to any other 
VMCS. Thus: “VMPTRLD X” always makes X current and active; “VMPTRLD Y” always makes X not current (because 
it makes Y current); VMLAUNCH makes the launch state of X “launched” if X was current and its launch state was 
“clear”; and VMCLEAR X always makes X inactive and not current and makes its launch state “clear”. 


The figure does not illustrate operations that do not modify the VMCS state relative to these parameters (e.g., 
execution of VMPTRLD X when X is already current). Note that VMCLEAR X makes X “inactive, not current, and 
clear,” even if X’s current state is not defined (e.g., even if X has not yet been initialized). See Section 23.11.3. 
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Figure 23-1. States of VMCS X 


Because a shadow VMCS (see Section 23.10) cannot be used for VM entry, the launch state of a shadow VMCS is 
not meaningful. Figure 23-1 does not illustrate all the ways in which a shadow VMCS may be made active. 


23.2 FORMAT OF THE VMCS REGION 


A VMCS region comprises up to 4-KBytes.! The format of a VMCS region is given in Table 23-1. 


Table 23-1. Format of the VMCS Region 


Byte Offset Contents 
0 Bits 30:0: VMCS revision identifier 
Bit 31: shadow-VMCS indicator (see Section 23.10) 
4 VMxX-abort indicator 
8 VMCS data (implementation-specific format) 


1. The exact size is implementation specific and can be determined by consulting the VMX capability MSR [A32_VMX_BASIC to deter- 
mine the size of the VMCS region (see Appendix A.1). 
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The first 4 bytes of the VMCS region contain the VMCS revision identifier at bits 30:0.1 Processors that maintain 
VMCS data in different formats (see below) use different VMCS revision identifiers. These identifiers enable soft- 
ware to avoid using a VMCS region formatted for one processor on a processor that uses a different format.? Bit 31 
of this 4-byte region indicates whether the VMCS is a shadow VMCS (see Section 23.10). 


Software should write the VMCS revision identifier to the VMCS region before using that region for a VMCS. The 
VMCS revision identifier is never written by the processor; VMPTRLD fails if its operand references a VMCS region 
whose VMCS revision identifier differs from that used by the processor. (VMPTRLD also fails if the shadow-VMCS 
indicator is 1 and the processor does not support the 1-setting of the “VMCS shadowing” VM-execution control; see 
Section 23.6.2) Software can discover the VMCS revision identifier that a processor uses by reading the VMX capa- 
bility MSR IA32_VMX_BASIC (see Appendix A.1). 


Software should clear or set the shadow-VMCS indicator depending on whether the VMCS is to be an ordinary 
VMCS or a shadow VMCS (see Section 23.10). VMPTRLD fails if the shadow-VMCS indicator is set and the processor 
does not support the 1-setting of the “VMCS shadowing” VM-execution control. Software can discover support for 
this setting by reading the VMX capability MSR IA32_VMX_PROCBASED_CTLS2 (see Appendix A.3.3). 


The next 4 bytes of the VMCS region are used for the VMX-abort indicator. The contents of these bits do not 
control processor operation in any way. A logical processor writes a non-zero value into these bits if a VMX abort 
occurs (see Section 26.7). Software may also write into this field. 


The remainder of the VMCS region is used for VMCS data (those parts of the VMCS that control VMX non-root 
operation and the VMxX transitions). The format of these data is implementation-specific. VMCS data are discussed 
in Section 23.3 through Section 23.9. To ensure proper behavior in VMX operation, software should maintain the 
VMCS region and related structures (enumerated in Section 23.11.4) in writeback cacheable memory. Future 
implementations may allow or require a different memory type?. Software should consult the VMX capability MSR 
IA32_VMX_BASIC (see Appendix A.1). 


23.3. ORGANIZATION OF VMCS DATA 


The VMCS data are organized into six logical groups: 


° Guest-state area. Processor state is saved into the guest-state area on VM exits and loaded from there on 
VM entries. 


* Host-state area. Processor state is loaded from the host-state area on VM exits. 


¢ VM-execution control fields. These fields control processor behavior in VMX non-root operation. They 
determine in part the causes of VM exits. 


¢ VM-exit control fields. These fields control VM exits. 
¢ VM-entry control fields. These fields control VM entries. 


¢ VM-exit information fields. These fields receive information on VM exits and describe the cause and the 
nature of VM exits. On some processors, these fields are read-only.* 


The VM-execution control fields, the VM-exit control fields, and the VM-entry control fields are sometimes referred 
to collectively as VMX controls. 


1. Earlier versions of this manual specified that the VMCS revision identifier was a 32-bit field. For all processors produced prior to this 
change, bit 31 of the VMCS revision identifier was O. 


2. Logical processors that use the same VMCS revision identifier use the same size for VMCS regions. 


3. Alternatively, software may map any of these regions or structures with the UC memory type. Doing so is strongly discouraged 
unless necessary as it will cause the performance of transitions using those structures to suffer significantly. In addition, the pro- 
cessor will continue to use the memory type reported in the VMX capability MSR [A32_VMX_BASIC with exceptions noted in Appen- 
dix A.1. 


4. Software can discover whether these fields can be written by reading the VMX capability MSR [A32_VMX_MISC (see Appendix A.6). 
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23.4 GUEST-STATE AREA 


This section describes fields contained in the guest-state area of the VMCS. VM entries load processor state from 
these fields and VM exits store processor state into these fields. See Section 25.3.2 and Section 26.3 for details. 


23.4.1 Guest Register State 


The following fields in the guest-state area correspond to processor registers: 


* Control registers CRO, CR3, and CR4 (64 bits each; 32 bits on processors that do not support Intel 64 archi- 
tecture). 


° Debug register DR7 (64 bits; 32 bits on processors that do not support Intel 64 architecture). 
* RSP, RIP, and RFLAGS (64 bits each; 32 bits on processors that do not support Intel 64 architecture).! 
° The following fields for each of the registers CS, SS, DS, ES, FS, GS, LDTR, and TR: 

— Selector (16 bits). 


— Base address (64 bits; 32 bits on processors that do not support Intel 64 architecture). The base-address 
fields for CS, SS, DS, and ES have only 32 architecturally-defined bits; nevertheless, the corresponding 
VMCS fields have 64 bits on processors that support Intel 64 architecture. 


— Segment limit (32 bits). The limit field is always a measure in bytes. 
— Access rights (32 bits). The format of this field is given in Table 23-2 and detailed as follows: 


¢ The low 16 bits correspond to bits 23:8 of the upper 32 bits of a 64-bit segment descriptor. While bits 
19:16 of code-segment and data-segment descriptors correspond to the upper 4 bits of the segment 
limit, the corresponding bits (bits 11:8) are reserved in this VMCS field. 


¢ Bit 16 indicates an unusable segment. Attempts to use such a segment fault except in 64-bit mode. 
In general, a segment register is unusable if it has been loaded with a null selector.2 


e Bits 31:17 are reserved. 


Table 23-2. Format of Access Rights 


Bit Position(s) Field 

3:0 Segment type 

4 S — Descriptor type (O = system; 1 = code or data) 
6:5 DPL — Descriptor privilege level 

7 P — Segment present 

11:8 Reserved 

12 AVL — Available for use by system software 


1. This chapter uses the notation RAX, RIP, RSP, RFLAGS, etc. for processor registers because most processors that support VMX oper- 
ation also support Intel 64 architecture. For processors that do not support Intel 64 architecture, this notation refers to the 32-bit 
forms of those registers (EAX, EIP, ESP, EFLAGS, etc.). In a few places, notation such as EAX is used to refer specifically to lower 32 
bits of the indicated register. 


2. There are a few exceptions to this statement. For example, a segment with a non-null selector may be unusable following a task 
switch that fails after its commit point; see “Interrupt 10—Invalid TSS Exception (#TS)” in Section 6.14, “Exception and Interrupt 
Handling in 64-bit Mode,” of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A. In contrast, the TR reg- 
ister is usable after processor reset despite having a null selector; see Table 10-1 in the Intel® 64 and IA-32 Architectures Software 
Developer's Manual, Volume 3A. 
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Table 23-2. Format of Access Rights (Contd.) 


Bit Position(s) Field 
13 Reserved (except for CS) 
L — 64-bit mode active (for CS only) 
14 D/B — Default operation size (0 = 16-bit segment; 1 = 32-bit segment) 
15 G — Granularity 
16 Segment unusable (0 = usable; 1 = unusable) 
31:17 Reserved 


The base address, segment limit, and access rights compose the “hidden” part (or “descriptor cache”) of each 
segment register. These data are included in the VMCS because it is possible for a segment register’s descriptor 
cache to be inconsistent with the segment descriptor in memory (in the GDT or the LDT) referenced by the 
segment register’s selector. 


The value of the DPL field for SS is always equal to the logical processor's current privilege level (CPL).4 


On some processors, executions of VMWRITE ignore attempts to write non-zero values to any of bits 11:8 or 
bits 31:17. On such processors, VMREAD always returns 0 for those bits, and VM entry treats those bits as if 
they were all 0 (see Section 25.3.1.2). 


The following fields for each of the registers GDTR and IDTR: 
— Base address (64 bits; 32 bits on processors that do not support Intel 64 architecture). 


— Limit (32 bits). The limit fields contain 32 bits even though these fields are specified as only 16 bits in the 
architecture. 


The following MSRs: 
— IA32_DEBUGCTL (64 bits) 
— IA32_SYSENTER_CS (32 bits) 


— JIA32_SYSENTER_ESP and IA32_SYSENTER_EIP (64 bits; 32 bits on processors that do not support Intel 64 
architecture) 


— IA32_PERF_GLOBAL_CTRL (64 bits). This field is supported only on processors that support the 1-setting 
of the “load IA32_PERF_GLOBAL_CTRL” VM-entry control. 


— IA32_PAT (64 bits). This field is supported only on processors that support either the 1-setting of the “load 
IA32_PAT” VM-entry control or that of the “save IA32_PAT” VM-exit control. 


— IA32_EFER (64 bits). This field is supported only on processors that support either the 1-setting of the “load 
IA32_EFER” VM-entry control or that of the “save IA32_EFER” VM-exit control. 


— IA32_BNDCFGS (64 bits). This field is supported only on processors that support either the 1-setting of the 
“load IA32_BNDCFGS” VM-entry control or that of the “clear IA32_BNDCFGS” VM-exit control. 


— IA32_RTIT_CTL (64 bits). This field is supported only on processors that support either the 1-setting of the 
“load IA32_RTIT_CTL” VM-entry control or that of the “clear IA32_RTIT_CTL” VM-exit control. 


— IA32_S_CET (64 bits; 32 bits on processors that do not support Intel 64 architecture). This field is 
supported only on processors that support the 1-setting of the “load CET state” VM-entry control. 


— IA32_INTERRUPT_SSP_TABLE_ADDR (64 bits; 32 bits on processors that do not support Intel 64 archi- 
tecture). This field is supported only on processors that support the 1-setting of the “load CET state” VM- 
entry control. 


— IA32_PKRS (64 bits). This field is supported only on processors that support the 1-setting of the “load 
PKRS” VM-entry control. 


. In protected mode, CPL is also associated with the RPL field in the CS selector. However, the RPL fields are not meaningful in real- 
address mode or in virtual-8086 mode. 
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The shadow-stack pointer register SSP (64 bits; 32 bits on processors that do not support Intel 64 archi- 
tecture). This field is supported only on processors that support the 1-setting of the “load CET state” VM-entry 
control. 


The register SMBASE (32 bits). This register contains the base address of the logical processor’s SMRAM image. 


23.4.2 Guest Non-Register State 


In addition to the register state described in Section 23.4.1, the guest-state area includes the following fields that 
characterize guest state but which do not correspond to processor registers: 


Activity state (32 bits). This field identifies the logical processor’s activity state. When a logical processor is 

executing instructions normally, it is in the active state. Execution of certain instructions and the occurrence 
of certain events may cause a logical processor to transition to an inactive state in which it ceases to execute 
instructions. 


The following activity states are defined: 


— 0: Active. The logical processor is executing instructions normally. 


1: HLT. The logical processor is inactive because it executed the HLT instruction. 


2: Shutdown. The logical processor is inactive because it incurred a triple fault? or some other serious 
error. 


— 3: Wait-for-SIPI. The logical processor is inactive because it is waiting for a startup-IPI (SIPI). 


Future processors may include support for other activity states. Software should read the VMX capability MSR 
IA32_VMX_MISC (see Appendix A.6) to determine what activity states are supported. 


Interruptibility state (32 bits). The IA-32 architecture includes features that permit certain events to be 
blocked for a period of time. This field contains information about such blocking. Details and the format of this 
field are given in Table 23-3. 


Table 23-3. Format of Interruptibility State 


Bit 
Position(s) 


Bit Name 


Notes 


0 


Blocking by STI 


See the “STI—Set Interrupt Flag” section in Chapter 4 of the Intel® 64 and IA-32 Architectures 
Software Developer's Manual, Volume 2B. 


Execution of STI with RFLAGS.IF = 0 blocks maskable interrupts on the instruction boundary 
following its execution.' Setting this bit indicates that this blocking is in effect. 


Blocking by 
MOV SS 


See Section 6.8.3, “Masking Exceptions and Interrupts When Switching Stacks,” in the Intel® 64 
and IA-32 Architectures Software Developer's Manual, Volume 3A. 


Execution of a MOV to SS or a POP to SS blocks or suppresses certain debug exceptions as well 
as interrupts (maskable and nonmaskable) on the instruction boundary following its execution. 
Setting this bit indicates that this blocking is in effect.2 This document uses the term “blocking 
by MOV SS,” but it applies equally to POP SS. 


Blocking by SMI 


See Section 30.2, “System Management Interrupt (SMI).” System-management interrupts 
(SMls) are disabled while the processor is in system-management mode (SMM). Setting this bit 
indicates that blocking of SMls is in effect. 


1. Execution of the MWAIT instruction may put a logical processor into an inactive state. However, this VMCS field never reflects this 
state. See Section 26.1. 


2. A triple fault occurs when a logical processor encounters an exception while attempting to deliver a double fault. 
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Table 23-3. Format of Interruptibility State (Contd.) 


Bit 
Position(s) 


Bit Name 


Notes 


3 


Blocking by NMI 


See Section 6.7.1, “Handling Multiple NMls,” in the Intel® 64 and IA-32 Architectures Software 
Developer's Manual, Volume 3A and Section 30.8, “NMI Handling While in SMM.” 


Delivery of a non-maskable interrupt (NMI) or a system-management interrupt (SMI) blocks 
subsequent NMls until the next execution of IRET. See Section 24.3 for how this behavior of 
IRET may change in VMX non-root operation. Setting this bit indicates that blocking of NMIs is 
in effect. Clearing this bit does not imply that NMIs are not (temporarily) blocked for other 
reasons. 

If the “virtual NMIs" VM-execution control (see Section 23.6.1) is 1, this bit does not control the 
blocking of NMIs. Instead, it refers to “virtual-NMI blocking” (the fact that guest software is not 
ready for an NMI). 


Enclave 
interruption 


Set to 1 if the VM exit occurred while the logical processor was in enclave mode. 


Such VM exits includes those caused by interrupts, non-maskable interrupts, system- 
management interrupts, INIT signals, and exceptions occurring in enclave mode as well as 
exceptions encountered during the delivery of such events incident to enclave mode. 


A VM exit that is incident to delivery of an event injected by VM entry leaves this bit 
unmodified. 


31:5 
NOTES: 


Reserved 


VM entry will fail if these bits are not 0. See Section 25.3.1.5. 


1. Nonmaskable interrupts and system-management interrupts may also be inhibited on the instruction boundary following such an 
execution of STI. 


2. System-management interrupts may also be inhibited on the instruction boundary following such an execution of MOV or POP. 


¢ Pending debug exceptions (64 bits; 32 bits on processors that do not support Intel 64 architecture). IA-32 
processors may recognize one or more debug exceptions without immediately delivering them.! This field 
contains information about such exceptions. This field is described in Table 23-4. 


Table 23-4. Format of Pending-Debug-Exceptions 


Bit Bit Name Notes 

Position(s) 

3:0 B3 - BO When set, each of these bits indicates that the corresponding breakpoint condition was met. 
Any of these bits may be set even if the corresponding enabling bit in DR7 is not set. 

11:4 Reserved VM entry fails if these bits are not 0. See Section 25.3.1.5. 

12 Enabled When set, this bit indicates that at least one data or I/O breakpoint was met and was enabled in 

breakpoint DR7. 

13 Reserved VM entry fails if this bit is not 0. See Section 25.3.1.5. 

14 BS When set, this bit indicates that a debug exception would have been triggered by single-step 
execution mode. 

15 Reserved VM entry fails if this bit is not 0. See Section 25.3.1.5. 


1. For example, execution of a MOV to SS or a POP to SS may inhibit some debug exceptions for one instruction. See Section 6.8.3 of 
Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 3A. In addition, certain events incident to an instruction 
(for example, an INIT signal) may take priority over debug traps generated by that instruction. See Table 6-2 in the Intel® 64 and 
IA-32 Architectures Software Developer's Manual, Volume 3A. 
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Table 23-4. Format of Pending-Debug-Exceptions (Contd.) 


Bit Bit Name Notes 
Position(s) 
16 RTM When set, this bit indicates that a debug exception (#DB) or a breakpoint exception (#BP) 


occurred inside an RTM region while advanced debugging of RTM transactional regions was 
enabled (see Section 16.3.7, “RTM-Enabled Debugger Support,” of Intel® 64 and IA-32 
Architectures Software Developer's Manual, Volume 1).' 


63:17 Reserved VM entry fails if these bits are not 0. See Section 25.3.1.5. Bits 63:32 exist only on processors 
that support Intel 64 architecture. 


NOTES: 


1. In general, the format of this field matches that of DR6. However, DR6 clears bit 16 to indicate an RTM-related exception, while this 
field sets the bit to indicate that condition. 


* VMCS link pointer (64 bits). If the “VMCS shadowing” VM-execution control is 1, the VMREAD and VMWRITE 
instructions access the VMCS referenced by this pointer (see Section 23.10). Otherwise, software should set 
this field to FFFFFFFF_FFFFFFFFH to avoid VM-entry failures (See Section 25.3.1.5). 


¢ VMX-preemption timer value (32 bits). This field is supported only on processors that support the 1-setting 
of the “activate VMX-preemption timer” VM-execution control. This field contains the value that the VMX- 
preemption timer will use following the next VM entry with that setting. See Section 24.5.1 and Section 25.7.4. 


° Page-directory-pointer-table entries (PDPTEs; 64 bits each). These four (4) fields (PDPTEO, PDPTE1, 
PDPTE2, and PDPTE3) are supported only on processors that support the 1-setting of the “enable EPT” VM- 
execution control. They correspond to the PDPTEs referenced by CR3 when PAE paging is in use (see Section 
4.4 in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). They are used only if 
the “enable EPT” VM-execution control is 1. 


* Guest interrupt status (16 bits). This field is supported only on processors that support the 1-setting of the 
“virtual-interrupt delivery” VM-execution control. It characterizes part of the guest’s virtual-APIC state and 
does not correspond to any processor or APIC registers. It comprises two 8-bit subfields: 


— Requesting virtual interrupt (RVI). This is the low byte of the guest interrupt status. The processor 
treats this value as the vector of the highest priority virtual interrupt that is requesting service. (The value 
0 implies that there is no such interrupt.) 


— Servicing virtual interrupt (SVI). This is the high byte of the guest interrupt status. The processor treats 
this value as the vector of the highest priority virtual interrupt that is in service. (The value 0 implies that 
there is no such interrupt. ) 


See Chapter 28 for more information on the use of this field. 


° PML index (16 bits). This field is supported only on processors that support the 1-setting of the “enable PML” 
VM-execution control. It contains the logical index of the next entry in the page-modification log. Because the 
page-modification log comprises 512 entries, the PML index is typically a value in the range 0-511. Details of 
the page-modification log and use of the PML index are given in Section 27.2.6. 


23.5 HOST-STATE AREA 

This section describes fields contained in the host-state area of the VMCS. As noted earlier, processor state is 
loaded from these fields on every VM exit (see Section 26.5). 

All fields in the host-state area correspond to processor registers: 

* CRO, CR3, and CR4 (64 bits each; 32 bits on processors that do not support Intel 64 architecture). 

° RSP and RIP (64 bits each; 32 bits on processors that do not support Intel 64 architecture). 


° Selector fields (16 bits each) for the segment registers CS, SS, DS, ES, FS, GS, and TR. There is no field in the 
host-state area for the LDTR selector. 
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¢ Base-address fields for FS, GS, TR, GDTR, and IDTR (64 bits each; 32 bits on processors that do not support 
Intel 64 architecture). 


¢ The following MSRs: 

— IA32_SYSENTER_CS (32 bits) 

— IA32_SYSENTER_ESP and IA32_SYSENTER_EIP (64 bits; 32 bits on processors that do not support Intel 64 
architecture). 

— IA32_PERF_GLOBAL_CTRL (64 bits). This field is supported only on processors that support the 1-setting 
of the “load IA32_PERF_GLOBAL_CTRL” VM-exit control. 

— IA32_PAT (64 bits). This field is supported only on processors that support the 1-setting of the “load 
IA32_PAT” VM-exit control. 


— IA32_EFER (64 bits). This field is supported only on processors that support the 1-setting of the “load 
IA32_EFER” VM-exit control. 


— IA32_S_CET (64 bits; 32 bits on processors that do not support Intel 64 architecture). This field is 
supported only on processors that support the 1-setting of the “load CET state” VM-exit control. 


— IA32_INTERRUPT_SSP_TABLE_ADDR (64 bits; 32 bits on processors that do not support Intel 64 archi- 
tecture). This field is supported only on processors that support the 1-setting of the “load CET state” VM- 
exit control. 


— IA32_PKRS (64 bits). This field is supported only on processors that support the 1-setting of the “load 
PKRS” VM-exit control. 


¢ The shadow-stack pointer register SSP (64 bits; 32 bits on processors that do not support Intel 64 archi- 
tecture). This field is supported only on processors that support the 1-setting of the “load CET state” VM-exit 
control. 


In addition to the state identified here, some processor state components are loaded with fixed values on every 
VM exit; there are no fields corresponding to these components in the host-state area. See Section 26.5 for details 
of how state is loaded on VM exits. 


23.6 VM-EXECUTION CONTROL FIELDS 


The VM-execution control fields govern VMX non-root operation. These are described in Section 23.6.1 through 
Section 23.6.8. 


23.6.1 Pin-Based VM-Execution Controls 


The pin-based VM-execution controls constitute a 32-bit vector that governs the handling of asynchronous events 
(for example: interrupts).! Table 23-5 lists the controls. See Chapter 26 for how these controls affect processor 
behavior in VMX non-root operation. 


1. Some asynchronous events cause VM exits regardless of the settings of the pin-based VM-execution controls (see Section 24.2). 
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Table 23-5. Definitions of Pin-Based VM-Execution Controls 


Bit Position(s) | Name Description 
0 External-interrupt | If this control is 1, external interrupts cause VM exits. Otherwise, they are delivered normally 
exiting through the guest interrupt-descriptor table (IDT). If this control is 1, the value of RFLAGS.IF 


does not affect interrupt blocking. 


3 NMI exiting If this control is 1, non-maskable interrupts (NMIs) cause VM exits. Otherwise, they are 
delivered normally using descriptor 2 of the IDT. This control also determines interactions 
between IRET and blocking by NMI (see Section 24.3). 


5 Virtual NMIs If this control is 1, NMIs are never blocked and the “blocking by NMI" bit (bit 3) in the 
interruptibility-state field indicates “virtual-NMI blocking” (see Table 23-3). This control also 
interacts with the “NMI-window exiting” VM-execution control (see Section 23.6.2). 


6 Activate VMX- If this control is 1, the VMX-preemption timer counts down in VMX non-root operation; see 
preemption timer | Section 24.5.1. A VM exit occurs when the timer counts down to zero; see Section 24.2. 

7 Process posted If this control is 1, the processor treats interrupts with the posted-interrupt notification vector 
interrupts (see Section 23.6.8) specially, updating the virtual-APIC page with posted-interrupt requests 


(see Section 28.6). 


All other bits in this field are reserved, some to 0 and some to 1. Software should consult the VMX capability MSRs 
IA32_VMX_PINBASED_CTLS and IA32_VMX_TRUE_PINBASED_CTLS (see Appendix A.3.1) to determine how to set 
reserved bits. Failure to set reserved bits properly causes subsequent VM entries to fail (see Section 25.2.1.1). 


The first processors to support the virtual-machine extensions supported only the 1-settings of bits 1, 2, and 4. The 
VMX capability MSR IA32_VMX_PINBASED_CTLS will always report that these bits must be 1. Logical processors 
that support the O-settings of any of these bits will support the VMX capability MSR 
IA32_VMX_TRUE_PINBASED_CTLS MSR, and software should consult this MSR to discover support for the 0- 
settings of these bits. Software that is not aware of the functionality of any one of these bits should set that bit to 1. 


23.6.2 Processor-Based VM-Execution Controls 


The processor-based VM-execution controls constitute three vectors that govern the handling of synchronous 
events, mainly those caused by the execution of specific instructions.! These are the primary processor-based 
VM-execution controls (32 bits), the secondary processor-based VM-execution controls (32 bits), and the 
tertiary VM-execution controls (64 bits). 


Table 23-6 lists the primary processor-based VM-execution controls. See Chapter 24 for more details of how these 
controls affect processor behavior in VMX non-root operation. 


Table 23-6. Definitions of Primary Processor-Based VM-Execution Controls 


Bit Position(s) | Name Description 

2 Interrupt-window _ | If this control is 1,a VM exit occurs at the beginning of any instruction if RFLAGS.IF = 1 and 
exiting there are no other blocking of interrupts (see Section 23.4.2). 

3 Use TSC offsetting | This control determines whether executions of RDTSC, executions of RDTSCP, and executions 


of RDMSR that read from the I[A32_TIME_STAMP_COUNTER MSR return a value modified by 
the TSC offset field (see Section 23.6.5 and Section 24.3). 


7 HLT exiting This control determines whether executions of HLT cause VM exits. 

9 INVLPG exiting This determines whether executions of INVLPG cause VM exits. 

10 MWAIT exiting This control determines whether executions of MWAIT cause VM exits. 
11 RDPMC exiting This control determines whether executions of RDPMC cause VM exits. 


1. Some instructions cause VM exits regardless of the settings of the processor-based VM-execution controls (see Section 24.1.2), as 
do task switches (see Section 24.2). 
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Table 23-6. Definitions of Primary Processor-Based VM-Execution Controls (Contd.) 


Bit Position(s) | Name Description 
12 RDTSC exiting This control determines whether executions of RDTSC and RDTSCP cause VM exits. 
15 CR3-load exiting In conjunction with the CR3-target controls (see Section 23.6.7), this control determines 
whether executions of MOV to CR3 cause VM exits. See Section 24.1.3. 
The first processors to support the virtual-machine extensions supported only the 1-setting 
of this control. 
16 CR3-store exiting | This control determines whether executions of MOV from CR3 cause VM exits. 
The first processors to support the virtual-machine extensions supported only the 1-setting 
of this control. 
17 Activate tertiary This control determines whether the tertiary processor-based VM-execution controls are 
controls used. If this control is 0, the logical processor operates as if all the tertiary processor-based 
\V/M-execution controls were also 0. 
19 CR8-load exiting This control determines whether executions of MOV to CR8 cause VM exits. 
20 CR8-store exiting | This control determines whether executions of MOV from CR8 cause VM exits. 
21 Use TPR shadow Setting this control to 1 enables TPR virtualization and other APIC-virtualization features. See 
Chapter 28. 
22 NMI-window If this control is 1,a VM exit occurs at the beginning of any instruction if there is no virtual- 
exiting NMI blocking (see Section 23.4.2). 
23 MOV-DR exiting This control determines whether executions of MOV DR cause VM exits. 
24 Unconditional I/O —_| This control determines whether executions of I/O instructions (IN, INS/INSB/INSW/INSD, OUT, 
exiting and OUTS/OUTSB/OUTSW/OUTSD) cause VM exits. 
25 Use I/O bitmaps This control determines whether I/O bitmaps are used to restrict executions of I/O instructions 
(see Section 23.6.4 and Section 24.1.3). 
For this control, “O” means “do not use I/O bitmaps” and “1” means “use I/O bitmaps.” If the I/O 
bitmaps are used, the setting of the “unconditional I/O exiting” control is ignored. 
27 Monitor trap flag If this control is 1, the monitor trap flag debugging feature is enabled. See Section 24.5.2. 
28 Use MSR bitmaps — | This control determines whether MSR bitmaps are used to control execution of the RDMSR 
and WRMSR instructions (see Section 23.6.9 and Section 24.1.3). 
For this control, “O” means “do not use MSR bitmaps” and “1” means “use MSR bitmaps.” If the 
MSR bitmaps are not used, all executions of the RDMSR and WRMSR instructions cause 
VM exits. 
29 MONITOR exiting This control determines whether executions of MONITOR cause VM exits. 
30 PAUSE exiting This control determines whether executions of PAUSE cause VM exits. 
31 Activate secondary | This control determines whether the secondary processor-based VM-execution controls are 
controls used. If this control is O, the logical processor operates as if all the secondary processor-based 
\V/M-execution controls were also 0. 


All other bits in this field are reserved, some to 0 and some to 1. Software should consult the VMX capability MSRs 
IA32_VMX_PROCBASED_CTLS and IA32_VMX_TRUE_PROCBASED_CTLS (see Appendix A.3.2) to determine how 
to set reserved bits. Failure to set reserved bits properly causes subsequent VM entries to fail (see Section 


25.2.1.1). 


The first processors to support the virtual-machine extensions supported only the 1-settings of bits 1, 4-6, 8, 13- 
16, and 26. The VMX capability MSR IA32_VMX_PROCBASED_CTLS will always report that these bits must be 1. 
Logical processors that support the O-settings of any of these bits will support the VMX capability MSR 
IA32_VMX_TRUE_PROCBASED_CTLS MSR, and software should consult this MSR to discover support for the 0- 


settings of these bits. Software that is not aware of the functionality of any one of these bits should set that bit to 1. 


Bit 31 of the primary processor-based VM-execution controls determines whether the secondary processor-based 
VM-execution controls are used. If that bit is 0, VM entry and VMX non-root operation function as if all the 
secondary processor-based VM-execution controls were 0. Processors that support only the O-setting of bit 31 of 
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the primary processor-based VM-execution controls do not support the secondary processor-based VM-execution 


controls. 


Table 23-7 lists the secondary processor-based VM-execution controls. See Chapter 24 for more details of how 
these controls affect processor behavior in VMX non-root operation. 


Table 23-7. Definitions of Secondary Processor-Based VM-Execution Controls 


Bit Position(s) | Name Description 
0 Virtualize APIC If this control is 1, the logical processor treats specially accesses to the page with the APIC- 
accesses access address. See Section 28.4. 
1 Enable EPT If this control is 1, extended page tables (EPT) are enabled. See Section 27.2. 
2 Descriptor-table This control determines whether executions of LGDT, LIDT, LLDT, LTR, SGDT, SIDT, SLDT, and 
exiting STR cause VM exits. 
3 Enable RDTSCP If this control is 0, any execution of RDTSCP causes an invalid-opcode exception (HUD). 
4 Virtualize x2APIC | If this control is 1, the logical processor treats specially RDMSR and WRMSR to APIC MSRs (in 
mode the range 800H-8FFH). See Section 28.5. 
5 Enable VPID If this control is 1, cached translations of linear addresses are associated with a virtual- 
processor identifier (VPID). See Section 27.1. 
6 WBINVD exiting This control determines whether executions of WBINVD and WBNOINVD cause VM exits. 
7 Unrestricted quest | This control determines whether quest software may run in unpaged protected mode or in real- 
address mode. 
8 APIC-register If this control is 1, the logical processor virtualizes certain APIC accesses. See Section 28.4 and 
virtualization Section 28.5. 
9 Virtual-interrupt This controls enables the evaluation and delivery of pending virtual interrupts as well as the 
delivery emulation of writes to the APIC registers that control interrupt prioritization. 
10 PAUSE-loop exiting | This control determines whether a series of executions of PAUSE can cause a VM exit (see 
Section 23.6.13 and Section 24.1.3). 
11 RDRAND exiting This control determines whether executions of RDRAND cause VM exits. 
12 Enable INVPCID If this control is 0, any execution of INVPCID causes a #UD. 
13 Enable Setting this control to 1 enables use of the VMFUNC instruction in VMX non-root operation. See 
VM functions Section 24.5.6. 
14 VMCS shadowing If this control is 1, executions of VMREAD and VMWRITE in VMX non-root operation may access 
a shadow VMCS (instead of causing VM exits). See Section 23.10 and Section 29.3. 
15 Enable ENCLS If this control is 1, executions of ENCLS consult the ENCLS-exiting bitmap to determine whether 
exiting the instruction causes a VM exit. See Section 23.6.16 and Section 24.1.3. 
16 RDSEED exiting This control determines whether executions of RDSEED cause VM exits. 
17 Enable PML If this control is 1, an access to a quest-physical address that sets an EPT dirty bit first adds an 
entry to the page-modification log. See Section 27.2.6. 
18 EPT-violation #VE | If this control is 1, EPT violations may cause virtualization exceptions (#VE) instead of VM exits. 
See Section 24.5.7. 
19 Conceal VMX from | If this control is 1, Intel Processor Trace suppresses from PIPs an indication that the processor 
PT was in VMX non-root operation and omits a VMCS packet from any PSB+ produced in VMX non- 
root operation (see Chapter 31). 
20 Enable If this control is 0, any execution of XSAVES or XRSTORS causes a #UD. 
XSAVES/XRSTORS 
22 Mode-based If this control is 1, EPT execute permissions are based on whether the linear address being 
execute control for | accessed is supervisor mode or user mode. See Chapter 27. 
EPT 
23 Sub-page write If this control is 1, EPT write permissions may be specified at the granularity of 128 bytes. See 
permissions for Section 27.2.4. 
EPT 
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Table 23-7. Definitions of Secondary Processor-Based VM-Execution Controls (Contd.) 
Bit Position(s) | Name Description 


24 Intel PT uses guest | If this control is 1, all output addresses used by Intel Processor Trace are treated as quest- 
physical addresses | physical addresses and translated using EPT. See Section 24.5.4. 


25 Use TSC scaling This control determines whether executions of RDTSC, executions of RDTSCP, and executions 
of RDMSR that read from the [A32_TIME_STAMP_COUNTER MSR return a value modified by the 
TSC multiplier field (see Section 23.6.5 and Section 24.3). 


26 Enable user wait If this control is O, any execution of TPAUSE, UMONITOR, or UMWAIT causes a #UD. 
and pause 

28 Enable ENCLV If this control is 1, executions of ENCLV consult the ENCLV-exiting bitmap to determine whether 
exiting the instruction causes a VM exit. See Section 23.6.17 and Section 24.1.3. 


All other bits in this field are reserved to 0. Software should consult the VMX capability MSR 
IA32_VMX_PROCBASED_CTLS2 (see Appendix A.3.3) to determine which bits may be set to 1. Failure to clear 
reserved bits causes subsequent VM entries to fail (see Section 25.2.1.1). 


Table 23-8 lists the tertiary processor-based VM-execution controls. See Chapter 24 for more details of how these 
controls affect processor behavior in VMX non-root operation. 


Table 23-8. Definitions of Tertiary Processor-Based VM-Execution Controls 
Bit Position(s) | Name Description 
0 LOADIWKEY exiting | This control determines whether executions of LOADIWKEY cause VM exits. 


All other bits in this field are reserved to 0. Software should consult the VMX capability MSR 
IA32_VMX_PROCBASED_CTLS3 (see Appendix A.3.4) to determine which bits may be set to 1. Failure to clear 
reserved bits causes subsequent VM entries to fail (see Section 25.2.1.1). 


23.6.3. Exception Bitmap 


The exception bitmap is a 32-bit field that contains one bit for each exception. When an exception occurs, its 
vector is used to select a bit in this field. If the bit is 1, the exception causes a VM exit. If the bit is 0, the exception 
is delivered normally through the IDT, using the descriptor corresponding to the exception’s vector. 


Whether a page fault (exception with vector 14) causes a VM exit is determined by bit 14 in the exception bitmap 
as well as the error code produced by the page fault and two 32-bit fields in the VMCS (the page-fault error-code 
mask and page-fault error-code match). See Section 24.2 for details. 


23.6.4  1/0-Bitmap Addresses 


The VM-execution control fields include the 64-bit physical addresses of I/O bitmaps A and B (each of which are 
4 KBytes in size). I/O bitmap A contains one bit for each I/O port in the range OOOOH through 7FFFH; I/O bitmap B 
contains bits for ports in the range 8000H through FFFFH. 


A logical processor uses these bitmaps if and only if the “use I/O bitmaps” control is 1. If the bitmaps are used, 
execution of an I/O instruction causes a VM exit if any bit in the I/O bitmaps corresponding to a port it accesses is 
1. See Section 24.1.3 for details. If the bitmaps are used, their addresses must be 4-KByte aligned. 


23.6.5 Time-Stamp Counter Offset and Multiplier 


The VM-execution control fields include a 64-bit TSC-offset field. If the “RDTSC exiting” control is O and the “use 
TSC offsetting” control is 1, this field controls executions of the RDTSC and RDTSCP instructions. It also controls 
executions of the RDMSR instruction that read from the IA32_TIME_STAMP_COUNTER MSR. For all of these, the 
value of the TSC offset is added to the value of the time-stamp counter, and the sum is returned to guest software 
in EDX:EAX. 
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Processors that support the 1-setting of the “use TSC scaling” control also support a 64-bit TSC-multiplier field. 
If this control is 1 (and the “RDTSC exiting” control is O and the “use TSC offsetting” control is 1), this field also 
affects the executions of the RDTSC, RDTSCP, and RDMSR instructions identified above. Specifically, the contents 
of the time-stamp counter is first multiplied by the TSC multiplier before adding the TSC offset. 


See Chapter 24 for a detailed treatment of the behavior of RDTSC, RDTSCP, and RDMSR in VMX non-root operation. 


23.6.6 Guest/Host Masks and Read Shadows for CRO and CR4 


VM-execution control fields include guest/host masks and read shadows for the CRO and CR4 registers. These 

fields control executions of instructions that access those registers (including CLTS, LMSW, MOV CR, and SMSW), 

They are 64 bits on processors that support Intel 64 architecture and 32 bits on processors that do not. 

In general, bits set to 1 in a guest/host mask correspond to bits “owned” by the host: 

* Guest attempts to set them (using CLTS, LMSW, or MOV to CR) to values differing from the corresponding bits 
in the corresponding read shadow cause VM exits. 

* Guest reads (using MOV from CR or SMSW) return values for these bits from the corresponding read shadow. 


Bits cleared to 0 correspond to bits “owned” by the guest; guest attempts to modify them succeed and guest reads 
return values for these bits from the control register itself. 


See Chapter 26 for details regarding how these fields affect VMX non-root operation. 


23.6.7. CR3-Target Controls 


The VM-execution control fields include a set of 4 CR3-target values and a CR3-target count. The CR3-target 
values each have 64 bits on processors that support Intel 64 architecture and 32 bits on processors that do not. The 
CR3-target count has 32 bits on all processors. 


An execution of MOV to CR3 in VMX non-root operation does not cause a VM exit if its source operand matches one 
of these values. If the CR3-target count is n, only the first n CR3-target values are considered; if the CR3-target 
count is 0, MOV to CR3 always causes a VM exit 


There are no limitations on the values that can be written for the CR3-target values. VM entry fails (see Section 
25.2) if the CR3-target count is greater than 4. 


Future processors may support a different number of CR3-target values. Software should read the VMX capability 
MSR IA32_VMX_MISC (see Appendix A.6) to determine the number of values supported. 


23.6.8 Controls for APIC Virtualization 


There are three mechanisms by which software accesses registers of the logical processor's local APIC: 


¢ If the local APIC is in xAPIC mode, it can perform memory-mapped accesses to addresses in the 4-KByte page 
referenced by the physical address in the IA32_APIC_BASE MSR (see Section 10.4.4, “Local APIC Status and 
Location” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A and Inte/® 64 
Architecture Processor Topology Enumeration). 


¢ If the local APIC is in x2APIC mode, it can accesses the local APIC’s registers using the RDMSR and WRMSR 
instructions (see Inte/® 64 Architecture Processor Topology Enumeration). 


° In 64-bit mode, it can access the local APIC’s task-priority register (TPR) using the MOV CR8 instruction. 


There are five processor-based VM-execution controls (see Section 23.6.2) that control such accesses. There are 
“use TPR shadow”, “virtualize APIC accesses”, “virtualize x2APIC mode”, “virtual-interrupt delivery”, and “APIC- 
register virtualization”. These controls interact with the following fields: 


° APIC-access address (64 bits). This field contains the physical address of the 4-KByte APIC-access page. 
If the “virtualize APIC accesses” VM-execution control is 1, access to this page may cause VM exits or be 
virtualized by the processor. See Section 28.4. 


1. If the local APIC does not support x2APIC mode, it is always in xAPIC mode. 
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The APIC-access address exists only on processors that support the 1-setting of the “virtualize APIC accesses” 
VM-execution control. 


Virtual-APIC address (64 bits). This field contains the physical address of the 4-KByte virtual-APIC page. 
The processor uses the virtual-APIC page to virtualize certain accesses to APIC registers and to manage virtual 
interrupts; see Chapter 28. 


Depending on the setting of the controls indicated earlier, the virtual-APIC page may be accessed by the 
following operations: 


— The MOV CR8 instructions (see Section 28.3). 


— Accesses to the APIC-access page if, in addition, the “virtualize APIC accesses” VM-execution control is 1 
(see Section 28.4). 


— The RDMSR and WRMSR instructions if, in addition, the value of ECX is in the range 800H-8FFH (indicating 
an APIC MSR) and the “virtualize x2APIC mode” VM-execution control is 1 (see Section 28.5). 


If the “use TPR shadow” VM-execution control is 1, VM entry ensures that the virtual-APIC address is 4-KByte 
aligned. The virtual-APIC address exists only on processors that support the 1-setting of the “use TPR shadow” 
VM-execution control. 


TPR threshold (32 bits). Bits 3:0 of this field determine the threshold below which bits 7:4 of VTPR (see 
Section 28.1.1) cannot fall. If the “virtual-interrupt delivery” VM-execution control is 0, a VM exit occurs after 
an operation (e.g., an execution of MOV to CR8) that reduces the value of those bits below the TPR threshold. 
See Section 28.1.2. 


The TPR threshold exists only on processors that support the 1-setting of the “use TPR shadow” VM-execution 
control. 


EOI-exit bitmap (4 fields; 64 bits each). These fields are supported only on processors that support the 1- 
setting of the “virtual-interrupt delivery” VM-execution control. They are used to determine which virtualized 
writes to the APIC’s EOI register cause VM exits: 


— EOI_EXITO contains bits for vectors from 0 (bit 0) to 63 (bit 63). 

— EOI_EXIT1 contains bits for vectors from 64 (bit 0) to 127 (bit 63). 
— EOI_EXIT2 contains bits for vectors from 128 (bit 0) to 191 (bit 63). 
— EOI_EXIT3 contains bits for vectors from 192 (bit 0) to 255 (bit 63). 
See Section 28.1.4 for more information on the use of this field. 


Posted-interrupt notification vector (16 bits). This field is supported only on processors that support the 1- 
setting of the “process posted interrupts” VM-execution control. Its low 8 bits contain the interrupt vector that 
is used to notify a logical processor that virtual interrupts have been posted. See Section 28.6 for more 
information on the use of this field. 


Posted-interrupt descriptor address (64 bits). This field is supported only on processors that support the 
1-setting of the “process posted interrupts” VM-execution control. It is the physical address of a 64-byte 
aligned posted interrupt descriptor. See Section 28.6 for more information on the use of this field. 


23.6.9 MSR-Bitmap Address 


On processors that support the 1-setting of the “use MSR bitmaps” VM-execution control, the VM-execution control 
fields include the 64-bit physical address of four contiguous MSR bitmaps, which are each 1-KByte in size. This 
field does not exist on processors that do not support the 1-setting of that control. The four bitmaps are: 


Read bitmap for low MSRs (located at the MSR-bitmap address). This contains one bit for each MSR address 
in the range QOOOOOOOH to OOOO1FFFH. The bit determines whether an execution of RDMSR applied to that 
MSR causes a VM exit. 


Read bitmap for high MSRs (located at the MSR-bitmap address plus 1024). This contains one bit for each 
MSR address in the range COOOO000H toCOO0iFFFH. The bit determines whether an execution of RDMSR 
applied to that MSR causes a VM exit. 
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° Write bitmap for low MSRs (located at the MSR-bitmap address plus 2048). This contains one bit for each 
MSR address in the range OOOO0000H to OOOO1FFFH. The bit determines whether an execution of WRMSR 
applied to that MSR causes a VM exit. 


°* Write bitmap for high MSRs (located at the MSR-bitmap address plus 3072). This contains one bit for each 
MSR address in the range COOOO000H toCOOO1iFFFH. The bit determines whether an execution of WRMSR 
applied to that MSR causes a VM exit. 


A logical processor uses these bitmaps if and only if the “use MSR bitmaps” control is 1. If the bitmaps are used, an 
execution of RDMSR or WRMSR causes a VM exit if the value of RCX is in neither of the ranges covered by the 
bitmaps or if the appropriate bit in the MSR bitmaps (corresponding to the instruction and the RCX value) is 1. See 
Section 24.1.3 for details. If the bitmaps are used, their address must be 4-KByte aligned. 


23.6.10 Executive-VMCS Pointer 


The executive-VMCS pointer is a 64-bit field used in the dual-monitor treatment of system-management interrupts 
(SMIs) and system-management mode (SMM). SMM VM exits save this field as described in Section 30.15.2. 
VM entries that return from SMM use this field as described in Section 30.15.4. 


23.6.11 Extended-Page-Table Pointer (EPTP) 


The extended-page-table pointer (EPTP) contains the address of the base of EPT PML4 table (see Section 
27.2.2), as well as other EPT configuration information. The format of this field is shown in Table 23-9. 


Table 23-9. Format of Extended-Page-Table Pointer 


Bit Field 
Position(s) 
2:0 EPT paging-structure memory type (see Section 27.2.7): 


O = Uncacheable (UC) 
6 = Write-back (WB) 


Other values are reserved.! 


5:3 This value is 1 less than the EPT page-walk length (see Section 27.2.2) 

6 Setting this control to 1 enables accessed and dirty flags for EPT (see Section 27.2.5)° 

7 Setting this control to 1 enables enforcement of access rights for supervisor shadow-stack pages (see Section 
27.2.3.2)3 

11:8 Reserved 

N-1:12 Bits N-1:12 of the physical address of the 4-KByte aligned EPT PML4 table* 

63:N Reserved 

NOTES: 


1. Software should read the VMX capability MSR I[A32_VMX_EPT_VPID_CAP (see Appendix A.10) to determine what EPT paging-struc- 
ture memory types are supported. 


2. Not all processors support accessed and dirty flags for EPT. Software should read the VMX capability MSR 
IA32_VMX_EPT_VPID_CAP (see Appendix A.10) to determine whether the processor supports this feature. 


3. Not all processors enforce access rights for shadow-stack pages. Software should read the VMX capability MSR 
IA32_VMX_EPT_VPID_CAP (see Appendix A.10) to determine whether the processor supports this feature. 


4. N is the physical-address width supported by the logical processor. Software can determine a processor's physical-address width by 
executing CPUID with 8OO00008H in EAX. The physical-address width is returned in bits 7:0 of EAX. 


The EPTP exists only on processors that support the 1-setting of the “enable EPT” VM-execution control. 
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23.6.12 Virtual-Processor Identifier (VPID) 


The virtual-processor identifier (VPID) is a 16-bit field. It exists only on processors that support the 1-setting 
of the “enable VPID” VM-execution control. See Section 27.1 for details regarding the use of this field. 


23.6.13 Controls for PAUSE-Loop Exiting 
On processors that support the 1-setting of the “PAUSE-loop exiting” VM-execution control, the VM-execution 
control fields include the following 32-bit fields: 


¢ PLE_Gap. Software can configure this field as an upper bound on the amount of time between two successive 
executions of PAUSE in a loop. 


¢ PLE_Window. Software can configure this field as an upper bound on the amount of time a guest is allowed to 
execute in a PAUSE loop. 


These fields measure time based on a counter that runs at the same rate as the timestamp counter (TSC). See 
Section 24.1.3 for more details regarding PAUSE-loop exiting. 


23.6.14 \WVM-Function Controls 


The VM-function controls constitute a 64-bit vector that governs use of the VMFUNC instruction in VMX non-root 
operation. This field is supported only on processors that support the 1-settings of both the “activate secondary 
controls” primary processor-based VM-execution control and the “enable VM functions” secondary processor- 
based VM-execution control. 


Table 23-10 lists the VM-function controls. See Section 24.5.6 for more details of how these controls affect 
processor behavior in VMX non-root operation. 


Table 23-10. Definitions of VM-Function Controls 
Bit Position(s) | Name Description 


0 EPTP switching The EPTP-switching VM function changes the EPT pointer to a value chosen from the EPTP list. 
See Section 24.5.6.3. 


All other bits in this field are reserved to 0. Software should consult the VMX capability MSR IA32_VMX_VMFUNC 
(see Appendix A.11) to determine which bits are reserved. Failure to clear reserved bits causes subsequent 
VM entries to fail (see Section 25.2.1.1). 


Processors that support the 1-setting of the “EPTP switching” VM-function control also support a 64-bit field called 
the EPTP-list address. This field contains the physical address of the 4-KByte EPTP list. The EPTP list comprises 
512 8-Byte entries (each an EPTP value) and is used by the EPTP-switching VM function (see Section 24.5.6.3). 


23.6.15 WMCS Shadowing Bitmap Addresses 


On processors that support the 1-setting of the “VMCS shadowing” VM-execution control, the VM-execution control 
fields include the 64-bit physical addresses of the VMREAD bitmap and the VMWRITE bitmap. Each bitmap is 4 
KBytes in size and thus contains 32 KBits. The addresses are the VMREAD-bitmap address and the VMWRITE- 
bitmap address. 


If the “VMCS shadowing” VM-execution control is 1, executions of VMREAD and VMWRITE may consult these 
bitmaps (see Section 23.10 and Section 29.3). 


23.6.16 ENCLS-Exiting Bitmap 


The ENCLS-exiting bitmap is a 64-bit field. If the “enable ENCLS exiting” VM-execution control is 1, execution of 
ENCLS causes a VM exit if the bit in this field corresponding to the value of EAX is 1. If the bit is 0, the instruction 
executes normally. See Section 24.1.3 for more information. 
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23.6.17 ENCLV-Exiting Bitmap 


The ENCLV-exiting bitmap is a 64-bit field. If the “enable ENCLV exiting” VM-execution control is 1, execution of 
ENCLV causes a VM exit if the bit in this field corresponding to the value of EAX is 1. If the bit is 0, the instruction 
executes normally. See Section 24.1.3 for more information. 


23.6.18 Control Field for Page-Modification Logging 


The PML address is a 64-bit field. It is the 4-KByte aligned address of the page-modification log. The page- 
modification log consists of 512 64-bit entries. It is used for the page-modification logging feature. Details of the 
page-modification logging are given in Section 27.2.6. 


If the “enable PML” VM-execution control is 1, VM entry ensures that the PML address is 4-KByte aligned. The PML 
address exists only on processors that support the 1-setting of the “enable PML” VM-execution control. 


23.6.19 Controls for Virtualization Exceptions 


On processors that support the 1-setting of the “EPT-violation #VE” VM-execution control, the VM-execution 
control fields include the following: 


* Virtualization-exception information address (64 bits). This field contains the physical address of the 
virtualization-exception information area. When a logical processor encounters a virtualization exception, 
it saves virtualization-exception information at the virtualization-exception information address; see Section 
24.5.7.2. 


¢ EPTP index (16 bits). When an EPT violation causes a virtualization exception, the processor writes the value 
of this field to the virtualization-exception information area. The EPTP-switching VM function updates this field 
(see Section 24.5.6.3). 


23.6.20 XSS-Exiting Bitmap 


On processors that support the 1-setting of the “enable XSAVES/XRSTORS” VM-execution control, the VM-execu- 
tion control fields include a 64-bit XSS-exiting bitmap. If the “enable XSAVES/XRSTORS” VM-execution control is 
1, executions of XSAVES and XRSTORS may consult this bitmap (see Section 24.1.3 and Section 24.3). 


23.6.21 Sub-Page-Permission-Table Pointer (SPPTP) 


If the sub-page write-permission feature of EPT is enabled, EPT write permissions may be determined at a 128- 
byte granularity (see Section 27.2.4). These permissions are determined using a hierarchy of sub-page-permission 
structures in memory. 


The root of this hierarchy is referenced by a VM-execution control field called the sub-page-permission-table 
pointer (SPPTP). The SPPTP contains the address of the base of the root SPP table (see Section 27.2.4.2). The 
format of this field is shown in Table 23-9. 


Table 23-11. Format of Sub-Page-Permission-Table Pointer 


Bit Field 

Position(s) 

11:0 Reserved 

N-1:12 Bits N-1:12 of the physical address of the 4-KByte aligned root SPP table 
63:N! Reserved 
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1. Nis the processor's physical-address width. Software can determine this width by executing CPUID with BOOOOOO8H in EAX. The 
physical-address width is returned in bits 7:0 of EAX. 


The SPPTP exists only on processors that support the 1-setting of the “sub-page write permissions for EPT” VM- 
execution control. 


ear 


VM-EXIT CONTROL FIELDS 


The VM-exit control fields govern the behavior of VM exits. They are discussed in Section 23.7.1 and Section 


23.7.2. 


23.7.1 


VM-Exit Controls 


The VM-exit controls constitute a 32-bit vector that governs the basic operation of VM exits. Table 23-12 lists the 
controls supported. See Chapter 26 for complete details of how these controls affect VM exits. 


Table 23-12. Definitions of VM-Exit Controls 


Bit Position(s) | Name Description 
2 Save debug controls | This control determines whether DR7 and the [A32_DEBUGCTL MSR are saved on VM exit. 
The first processors to support the virtual-machine extensions supported only the 1- 
setting of this control. 
9 Host address-space On processors that support Intel 64 architecture, this control determines whether a logical 
size processor is in 64-bit mode after the next VM exit. Its value is loaded into CS.L, 
IA32_EFER.LME, and |A32_EFER.LMA on every VM exit.! 
This control must be O on processors that do not support Intel 64 architecture. 
12 Load This control determines whether the [A32_PERF_GLOBAL_CTRL MSR is loaded on VM exit. 
IA32_PERF_GLOBAL_ 
CTRL 
15 Acknowledge This control affects VM exits due to external interrupts: 
interrupt on exit * If such a VM exit occurs and this control is 1, the logical processor acknowledges the 
interrupt controller, acquiring the interrupt’s vector. The vector is stored in the VM-exit 
interruption-information field, which is marked valid. 
« If sucha VM exit occurs and this control is 0, the interrupt is not acknowledged and the 
VM-exit interruption-information field is marked invalid. 
18 Save IA32_PAT This control determines whether the [A32_PAT MSR is saved on VM exit. 
19 Load |A32_PAT This control determines whether the [A32_PAT MSR is loaded on VM exit. 
20 Save IA32_EFER This control determines whether the |[A32_EFER MSR is saved on VM exit. 
21 Load IA32_EFER This control determines whether the [A32_EFER MSR is loaded on VM exit. 
22 Save VMX- This control determines whether the value of the VMX-preemption timer is saved on 
preemption timer VM exit. 
value 
23 Clear |A32_BNDCFGS | This control determines whether the [A32_BNDCFGS MSR is cleared on VM exit. 
24 Conceal VMX from PT | If this control is 1, Intel Processor Trace does not produce a paging information packet (PIP) 
ona VM exit or a VMCS packet on an SMM VM exit (see Chapter 31). 
25 Clear IA32_RTIT_CTL | This control determines whether the I[A32_RTIT_CTL MSR is cleared on VM exit. 
28 Load CET state This control determines whether CET-related MSRs and SPP are loaded on VM exit. 
29 Load PKRS This control determines whether the [A32_PKRS MSR is loaded on VM exit. 
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NOTES: 


1. Since the Intel 64 architecture specifies that IA32_EFER.LMA is always set to the logical-AND of CRO.PG and IA32_EFER.LME, and 
since CRO.PG is always 1 in VMX root operation, IA32_EFER.LMA is always identical to IA32_EFER.LME in VMX root operation. 


All other bits in this field are reserved, some to 0 and some to 1. Software should consult the VMX capability MSRs 
IA32_VMX_EXIT_CTLS and IA32_VMX_TRUE_EXIT_CTLS (see Appendix A.4) to determine how it should set the 
reserved bits. Failure to set reserved bits properly causes subsequent VM entries to fail (see Section 25.2.1.2). 


The first processors to support the virtual-machine extensions supported only the 1-settings of bits 0-8, 10, 11, 
13, 14, 16, and 17. The VMX capability MSR IA32_VMX_EXIT_CTLS always reports that these bits must be 1. 
Logical processors that support the O-settings of any of these bits will support the VMX capability MSR 
IA32_VMX_TRUE_EXIT_CTLS MSR, and software should consult this MSR to discover support for the 0-settings of 
these bits. Software that is not aware of the functionality of any one of these bits should set that bit to 1. 


23.7.2 VM-Exit Controls for MSRs 


AVMM may specify lists of MSRs to be stored and loaded on VM exits. The following VM-exit control fields deter- 
mine how MSRs are stored on VM exits: 


° VM-exit MSR-store count (32 bits). This field specifies the number of MSRs to be stored on VM exit. It is 
recommended that this count not exceed 512.1 Otherwise, unpredictable processor behavior (including a 
machine check) may result during VM exit. 


¢ VM-exit MSR-store address (64 bits). This field contains the physical address of the VM-exit MSR-store area. 
The area is a table of entries, 16 bytes per entry, where the number of entries is given by the VM-exit MSR-store 
count. The format of each entry is given in Table 23-13. If the VM-exit MSR-store count is not zero, the address 
must be 16-byte aligned. 


Table 23-13. Format of an MSR Entry 


Bit Position(s) Contents 
31:0 MSR index 
63:32 Reserved 
127:64 MSR data 


See Section 26.4 for how this area is used on VM exits. 
The following VM-exit control fields determine how MSRs are loaded on VM exits: 


° VM-exit MSR-load count (32 bits). This field contains the number of MSRs to be loaded on VM exit. It is 
recommended that this count not exceed 512. Otherwise, unpredictable processor behavior (including a 
machine check) may result during VM exit.2 


° VM-exit MSR-load address (64 bits). This field contains the physical address of the VM-exit MSR-load area. 
The area is a table of entries, 16 bytes per entry, where the number of entries is given by the VM-exit MSR-load 
count (see Table 23-13). If the VM-exit MSR-load count is not zero, the address must be 16-byte aligned. 


See Section 26.6 for how this area is used on VM exits. 


23.8 VM-ENTRY CONTROL FIELDS 


The VM-entry control fields govern the behavior of VM entries. They are discussed in Sections 23.8.1 through 
23.8.3. 


1. Future implementations may allow more MSRs to be stored reliably. Software should consult the VMX capability MSR 
IA32_V/MX_MISC to determine the number supported (see Appendix A.6). 


2. Future implementations may allow more MSRs to be loaded reliably. Software should consult the VMX capability MSR 
IA32_VMX_MISC to determine the number supported (see Appendix A.6). 
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VM-Entry Controls 


The VM-entry controls constitute a 32-bit vector that governs the basic operation of VM entries. Table 23-14 lists 
the controls supported. See Chapter 23 for how these controls affect VM entries. 


Table 23-14. Definitions of VM-Entry Controls 


Bit Position(s) | Name Description 
2 Load debug This control determines whether DR7 and the IA32_DEBUGCTL MSR are loaded on VM entry. 
controls The first processors to support the virtual-machine extensions supported only the 1-setting of 
this control. 
9 IA-32e mode guest | On processors that support Intel 64 architecture, this control determines whether the logical 
processor is in |A-32e mode after VM entry. Its value is loaded into IA32_EFER.LMA as part of 
VM entry.! 
This control must be O on processors that do not support Intel 64 architecture. 
10 Entry to SMM This control determines whether the logical processor is in system-management mode (SMM) 
after VM entry. This control must be O for any VM entry from outside SMM. 
11 Deactivate dual- If set to 1, the default treatment of SMIs and SMM is in effect after the VM entry (see Section 
monitor treatment | 30.15.7). This control must be O for any VM entry from outside SMM. 
13 Load This control determines whether the IA32_PERF_GLOBAL_CTRL MSR is loaded on VM entry. 
IA32_PERF_GLOBA 
L_CTRL 
14 Load |A32_PAT This control determines whether the [A32_PAT MSR is loaded on VM entry. 
15 Load IA32_EFER This control determines whether the I[A32_EFER MSR is loaded on VM entry. 
16 Load This control determines whether the [A32_BNDCFGS MSR is loaded on VM entry. 
IA32_BNDCFGS 
17 Conceal VMX from _ | If this control is 1, Intel Processor Trace does not produce a paging information packet (PIP) on 
PT a VM entry or a VMCS packet on a VM entry that returns from SMM (see Chapter 31). 
18 Load This control determines whether the IA32_RTIT_CTL MSR is loaded on VM entry. 
IA32_RTIT_CTL 
20 Load CET state This control determines whether CET-related MSRs and SPP are loaded on VM entry. 
22 Load PKRS This control determines whether the IA32_PKRS MSR is loaded on VM entry. 
NOTES: 


1. Bit 5 of the [A32_VMX_MISC MSR is read as 1 on any logical processor that supports the 1-setting of the “unrestricted quest” VM- 
execution control. If it is read as 1, every VM exit stores the value of [A32_EFER.LMA into the “IA-32e mode quest” VM-entry control 
(see Section 26.2). 


All other bits in this field are reserved, some to 0 and some to 1. Software should consult the VMX capability MSRs 
IA32_VMX_ENTRY_CTLS and IA32_VMX_TRUE_ENTRY_CTLS (see Appendix A.5) to determine how it should set 
the reserved bits. Failure to set reserved bits properly causes subsequent VM entries to fail (see Section 25.2.1.3). 


The first processors to support the virtual-machine extensions supported only the 1-settings of bits O-8 and 12. 
The VMX capability MSR IA32_VMX_ENTRY_CTLS always reports that these bits must be 1. Logical processors that 
support the O-settings of any of these bits will support the VMX capability MSR IA32_VMX_TRUE_ENTRY_CTLS 
MSR, and software should consult this MSR to discover support for the 0-settings of these bits. Software that is not 
aware of the functionality of any one of these bits should set that bit to 1. 


23.8.2 


VM-Entry Controls for MSRs 


A VMM may specify a list of MSRs to be loaded on VM entries. The following VM-entry control fields manage this 


functionality: 


Vol. 3C 23-21 


VIRTUAL MACHINE CONTROL STRUCTURES 


VM-entry MSR-load count (32 bits). This field contains the number of MSRs to be loaded on VM entry. It is 
recommended that this count not exceed 512. Otherwise, unpredictable processor behavior (including a 
machine check) may result during VM entry. 

VM-entry MSR-load address (64 bits). This field contains the physical address of the VM-entry MSR-load 


area. The area is a table of entries, 16 bytes per entry, where the number of entries is given by the VM-entry 
MSR-load count. The format of entries is described in Table 23-13. If the VM-entry MSR-load count is not zero, 


the address must be 16-byte aligned. 


See Section 25.4 for details of how this area is used on VM entries. 


23.8.3. VM-Entry Controls for Event Injection 


VM entry can be configured to conclude by delivering an event through the IDT (after all guest state and MSRs have 
been loaded). This process is called event injection and is controlled by the following three VM-entry control 


fields: 


° VM-entry interruption-information field (32 bits). This field provides details about the event to be injected. 
Table 23-15 describes the field. 


Table 23-15. Format of the VM-Entry Interruption-Information Field 


Bit Position(s) | Content 
7:0 Vector of interrupt or exception 
10:8 Interruption type: 
0: External interrupt 
1: Reserved 
2: Non-maskable interrupt (NMI) 
3: Hardware exception (e.g,. #PF) 
4: Software interrupt (INT n) 
5: Privileged software exception (INT1) 
6: Software exception (INT3 or INTO) 
7: Other event 
11 Deliver error code (0 = do not deliver; 1 = deliver) 
30:12 Reserved 
31 Valid 


— The vector (bits 7:0) determines which entry in the IDT is used or which other event is injected. 


— The interruption type (bits 10:8) determines details of how the injection is performed. In general, a VMM 
should use the type hardware exception for all exceptions other than the following: 


¢ breakpoint exceptions (#BP; a VMM should use the type software exception); 
* overflow exceptions (#OF a VMM should use the use type software exception); and 


¢ those debug exceptions (#DB) that are generated by INT1 (a VMM should use the use type privileged 
software exception). 


The type other event is used for injection of events that are not delivered through the IDT.? 


— For exceptions, the deliver-error-code bit (bit 11) determines whether delivery pushes an error code on 
the guest stack. 


— VM entry injects an event if and only if the valid bit (bit 31) is 1. The valid bit in this field is cleared on every 
VM exit (see Section 26.2). 


. Future implementations may allow more MSRs to be loaded reliably. Software should consult the VMX capability MSR 
IA32_VMX_MISC to determine the number supported (See Appendix A.6). 


. The type hardware exception should be used for all other debug exceptions. 
INT1 and INT3 refer to the instructions with opcodes F1 and CC, respectively, and not to INT n with values 1 or 3 for n. 
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¢ VM-entry exception error code (32 bits). This field is used if and only if the valid bit (bit 31) and the deliver- 
error-code bit (bit 11) are both set in the VM-entry interruption-information field. 


¢ VM-entry instruction length (32 bits). For injection of events whose type is software interrupt, software 
exception, or privileged software exception, this field is used to determine the value of RIP that is pushed on 
the stack. 


See Section 25.6 for details regarding the mechanics of event injection, including the use of the interruption type 
and the VM-entry instruction length. 


VM exits clear the valid bit (bit 31) in the VM-entry interruption-information field. 


23.9 VM-EXIT INFORMATION FIELDS 


The VMCS contains a section of fields that contain information about the most recent VM exit. 


On some processors, attempts to write to these fields with VMWRITE fail (see “VMWRITE—Write Field to Virtual- 
Machine Control Structure” in Chapter 29).+ 


23.9.1 Basic VM-Exit Information 


The following VM-exit information fields provide basic information about a VM exit: 
¢ Exit reason (32 bits). This field encodes the reason for the VM exit and has the structure given in Table 23-16. 


Table 23-16. Format of Exit Reason 


Bit Position(s) Contents 

15:0 Basic exit reason 

16 Always cleared to 0 

26:17 Not currently defined 

27 A VM exit saves this bit as 1 to indicate that the VM exit was incident to enclave mode. 
28 Pending MTF VM exit 

29 VM exit from VMX root operation 

30 Not currently defined 

31 VM-entry failure (O = true VM exit; 1 = VM-entry failure) 


— Bits 15:0 provide basic information about the cause of the VM exit (if bit 31 is clear) or of the VM-entry 
failure (if bit 31 is set). Appendix C enumerates the basic exit reasons. 


— Bit 16 is always cleared to 0. 
— Bit 27 is set to 1 if the VM exit occurred while the logical processor was in enclave mode. 


A VM exit also sets this bit if it is incident to delivery of an event injected by VM entry and the guest inter- 
ruptibility-state field indicates an enclave interrupt (bit 4 of the field is 1). See Section 26.2.1 for details. 


— Bit 28 is set only by an SMM VM exit (see Section 30.15.2) that took priority over an MTF VM exit (see 
Section 24.5.2) that would have occurred had the SMM VM exit not occurred. See Section 30.15.2.3. 


— Bit 29 is set if and only if the processor was in VMX root operation at the time the VM exit occurred. This can 
happen only for SMM VM exits. See Section 30.15.2. 


1. Software can discover whether these fields can be written by reading the VMX capability MSR IA32_VMX_MISC (see Appendix A.6). 
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— Because some VM-entry failures load processor state from the host-state area (see Section 25.8), software 


must be able to distinguish such cases from true VM exits. Bit 31 is used for that purpose. 


° Exit qualification (64 bits; 32 bits on processors that do not support Intel 64 architecture). This field contains 


additional information about the cause of VM exits due to the following: debug exceptions; page-fault 
exceptions; start-up IPIs (SIPIs); task switches; INVEPT; INVLPG;INVVPID; LGDT; LIDT; LLDT; LTR; SGDT; 


SIDT; SLDT; STR; VMCLEAR; VMPTRLD; VMPTRST; VMREAD; VMWRITE; VMXON; XRSTORS; XSAVES; control- 
register accesses; MOV DR; I/O instructions; and MWAIT. The format of the field depends on the cause of the 


VM exit. See Section 26.2.1 for details. 


* Guest-linear address (64 bits; 32 bits on processors that do not support Intel 64 architecture). This field is 
used in the following cases: 


— VM exits due to attempts to execute LMSW with a memory operand. 
— VM exits due to attempts to execute INS or OUTS. 


— VM exits due to system-management interrupts (SMIs) that arrive immediately after retirement of I/O 
instructions. 


— Certain VM exits due to EPT violations 
See Section 26.2.1 and Section 30.15.2.3 for details of when and how this field is used. 


° Guest-physical address (64 bits). This field is used VM exits due to EPT violations and EPT misconfigurations. 


See Section 26.2.1 for details of when and how this field is used. 


23.9.2 Information for VM Exits Due to Vectored Events 


Event-specific information is provided for VM exits due to the following vectored events: exceptions (including 


those generated by the instructions INT3, INTO, INT1, BOUND, UDO, UD1, and UD2); external interrupts that occur 
while the “acknowledge interrupt on exit” VM-exit control is 1; and non-maskable interrupts (NMIs). This informa- 


tion is provided in the following fields: 


° \VM-exit interruption information (32 bits). This field receives basic information associated with the event 
causing the VM exit. Table 23-17 describes this field. 


Table 23-17. Format of the VM-Exit Interruption-Information Field 


Bit Position(s) | Content 


7:0 Vector of interrupt or exception 
10:8 Interruption type: 

0: External interrupt 

1: Not used 


2: Non-maskable interrupt (NMI) 
3: Hardware exception 

4: Not used 

5: Privileged software exception 
6: Software exception 


7: Not used 
11 Error code valid (0 = invalid; 1 = valid) 
12 NMI unblocking due to IRET 
30:13 Not currently defined 
31 Valid 


° \VM-exit interruption error code (32 bits). For VM exits caused by hardware exceptions that would have 
delivered an error code on the stack, this field receives that error code. 


Section 26.2.2 provides details of how these fields are saved on VM exits. 
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23.9.3 Information for VM Exits That Occur During Event Delivery 


Additional information is provided for VM exits that occur during event delivery in VMX non-root operation.! This 
information is provided in the following fields: 


¢ IDT-vectoring information (32 bits). This field receives basic information associated with the event that was 
being delivered when the VM exit occurred. Table 23-18 describes this field. 


Table 23-18. Format of the IDT-Vectoring Information Field 


Bit Position(s) | Content 


7:0 Vector of interrupt or exception 
10:8 Interruption type: 

0: External interrupt 

1: Not used 


2: Non-maskable interrupt (NMI) 
3: Hardware exception 

4: Software interrupt 

5: Privileged software exception 
6: Software exception 


7: Not used 
11 Error code valid (O = invalid; 1 = valid) 
30:12 Not currently defined 
31 Valid 


¢ IDT-vectoring error code (32 bits). For VM exits the occur during delivery of hardware exceptions that would 
have delivered an error code on the stack, this field receives that error code. 


See Section 26.2.4 provides details of how these fields are saved on VM exits. 


23.9.4 Information for VM Exits Due to Instruction Execution 
The following fields are used for VM exits caused by attempts to execute certain instructions in VMX non-root oper- 
ation: 


¢ VM-exit instruction length (32 bits). For VM exits resulting from instruction execution, this field receives the 
length in bytes of the instruction whose execution led to the VM exit.2 See Section 26.2.5 for details of when 
and how this field is used. 

° VM-exit instruction information (32 bits). This field is used for VM exits due to attempts to execute INS, 
INVEPT, INVVPID, LIDT, LGDT, LLDT, LTR, OUTS, SIDT, SGDT, SLDT, STR, VMCLEAR, VMPTRLD, VMPTRST, 
VMREAD, VMWRITE, or VMXON.? The format of the field depends on the cause of the VM exit. See Section 
26.2.5 for details. 

The following fields (64 bits each; 32 bits on processors that do not support Intel 64 architecture) are used only for 

VM exits due to SMIs that arrive immediately after retirement of I/O instructions. They provide information about 

that I/O instruction: 


¢ I/O RCX. The value of RCX before the I/O instruction started. 

¢ I/O RSI. The value of RSI before the I/O instruction started. 

¢ I/O RDI. The value of RDI before the I/O instruction started. 

¢ I/O RIP. The value of RIP before the I/O instruction started (the RIP that addressed the I/O instruction). 


1. This includes cases in which the event delivery was caused by event injection as part of VM entry; see Section 25.6.1.2. 
2. This field is also used for VM exits that occur during the delivery of a software interrupt or software exception. 


3. Whether the processor provides this information on VM exits due to attempts to execute INS or OUTS can be determined by con- 
sulting the VMX capability MSR [A32_VMX_BASIC (see Appendix A.1). 
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23.9.5 VM-Instruction Error Field 


The 32-bit VM-instruction error field does not provide information about the most recent VM exit. In fact, it is 
not modified on VM exits. Instead, it provides information about errors encountered by a non-faulting execution of 
one of the VMX instructions. 


23.10 VMCS TYPES: ORDINARY AND SHADOW 


Every VMCS is either an ordinary VMCS or a shadow VMCS. A VMCS's type is determined by the shadow-VMCS 
indicator in the VMCS region (this is the value of bit 31 of the first 4 bytes of the VMCS region; see Table 23-1): 0 
indicates an ordinary VMCS, while 1 indicates a shadow VMCS. Shadow VMCSs are supported only on processors 
that support the 1-setting of the “VMCS shadowing” VM-execution control (see Section 23.6.2). 


A shadow VMCS differs from an ordinary VMCS in two ways: 


¢ Anordinary VMCS can be used for VM entry but a shadow VMCS cannot. Attempts to perform VM entry when 
the current VMCS is a shadow VMCS fail (see Section 25.1). 


¢ The VMREAD and VMWRITE instructions can be used in VMX non-root operation to access a shadow VMCS but 
not an ordinary VMCS. This fact results from the following: 


— Ifthe “VMCS shadowing” VM-execution control is 0, execution of the VMREAD and VMWRITE instructions in 
VMX non-root operation always cause VM exits (see Section 24.1.3). 


— Ifthe “VMCS shadowing” VM-execution control is 1, execution of the VMREAD and VMWRITE instructions in 
VMX non-root operation can access the VMCS referenced by the VMCS link pointer (see Section 29.3). 


— If the“VMCS shadowing” VM-execution control is 1, VM entry ensures that any VMCS referenced by the 
VMCS link pointer is a shadow VMCS (see Section 25.3.1.5). 


In VMX root operation, both types of VMCSs can be accessed with the VMREAD and VMWRITE instructions. 


Software should not modify the shadow-VMCS indicator in the VMCS region of a VMCS that is active. Doing so may 
cause the VMCS to become corrupted (see Section 23.11.1). Before modifying the shadow-VMCS indicator, soft- 
ware should execute VMCLEAR for the VMCS to ensure that it is not active. 


23.11 SOFTWARE USE OF THE VMCS AND RELATED STRUCTURES 


This section details guidelines that software should observe when using a VMCS and related structures. It also 
provides descriptions of consequences for failing to follow guidelines. 


23.11.1 Software Use of Virtual-Machine Control Structures 
To ensure proper processor behavior, software should observe certain guidelines when using an active VMCS. 


No VMCS should ever be active on more than one logical processor. If a VMCS is to be “migrated” from one logical 
processor to another, the first logical processor should execute VMCLEAR for the VMCS (to make it inactive on that 
logical processor and to ensure that all VMCS data are in memory) before the other logical processor executes 
VMPTRLD for the VMCS (to make it active on the second logical processor).! A VMCS that is made active on more 
than one logical processor may become corrupted (see below). 


Software should not modify the shadow-VMCS indicator (see Table 23-1) in the VMCS region of a VMCS that is 
active. Doing so may cause the VMCS to become corrupted. Before modifying the shadow-VMCS indicator, software 
should execute VMCLEAR for the VMCS to ensure that it is not active. 


Software should use the VMREAD and VMWRITE instructions to access the different fields in the current VMCS (see 
Section 23.11.2). Software should never access or modify the VMCS data of an active VMCS using ordinary 


1. As noted in Section 23.1, execution of the VMPTRLD instruction makes a VMCS is active. In addition, VM entry makes active any 
shadow VMCS referenced by the VMCS link pointer in the current VMCS. If a shadow VMCS is made active by VM entry, it is neces- 
sary to execute VMCLEAR for that VMCS before allowing that VMCS to become active on another logical processor. 
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memory operations, in part because the format used to store the VMCS data is implementation-specific and not 
architecturally defined, and also because a logical processor may maintain some VMCS data of an active VMCS on 
the processor and not in the VMCS region. The following items detail some of the hazards of accessing VMCS data 
using ordinary memory operations: 


¢ Any data read from a VMCS with an ordinary memory read does not reliably reflect the state of the VMCS. 
Results may vary from time to time or from logical processor to logical processor. 


¢ Writing to a VMCS with an ordinary memory write is not guaranteed to have a deterministic effect on the VMCS. 
Doing so may cause the VMCS to become corrupted (see below). 


(Software can avoid these hazards by removing any linear-address mappings to a VMCS region before executing a 
VMPTRLD for that region and by not remapping it until after executing VMCLEAR for that region.) 


If a logical processor leaves VMX operation, any VMCSs active on that logical processor may be corrupted (see 
below). To prevent such corruption of a VMCS that may be used either after a return to VMX operation or on 
another logical processor, software should execute VMCLEAR for that VMCS before executing the VMXOFF instruc- 
tion or removing power from the processor (é.g., aS part of a transition to the S3 and S4 power states). 


This section has identified operations that may cause a VMCS to become corrupted. These operations may cause 
the VMCS's data to become undefined. Behavior may be unpredictable if that VMCS used subsequently on any 
logical processor. The following items detail some hazards of VMCS corruption: 


¢ VM entries may fail for unexplained reasons or may load undesired processor state. 


° The processor may not correctly support VMX non-root operation as documented in Chapter 24 and may 
generate unexpected VM exits. 


¢ VM exits may load undesired processor state, save incorrect state into the VMCS, or cause the logical processor 
to transition to a shutdown state. 


23.11.2 VMREAD, VMWRITE, and Encodings of VMCS Fields 


Every field of the VMCS is associated with a 32-bit value that is its encoding. The encoding is provided in an 
operand to VMREAD and VMWRITE when software wishes to read or write that field. These instructions fail if given, 
in 64-bit mode, an operand that sets an encoding bit beyond bit 32. See Chapter 29 for a description of these 
instructions. 


The structure of the 32-bit encodings of the VMCS components is determined principally by the width of the fields 
and their function in the VMCS. See Table 23-19. 


Table 23-19. Structure of VMCS Component Encoding 


Bit Position(s) Contents 
0 Access type (0 = full; 1 = high); must be full for 16-bit, 32-bit, and natural-width fields 
9:1 Index 
11:10 Type: 
0: control 


1: VM-exit information 
2: guest state 
3: host state 


12 Reserved (must be 0) 
14:13 Width: 

0: 16-bit 

1: 64-bit 

2: 32-bit 


3: natural-width 


31:15 Reserved (must be 0) 
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The following items detail the meaning of the bits in each encoding: 


Field width. Bits 14:13 encode the width of the field. 
— A value of 0 indicates a 16-bit field. 
— A value of 1 indicates a 64-bit field. 
— A value of 2 indicates a 32-bit field. 


— A value of 3 indicates a natural-width field. Such fields have 64 bits on processors that support Intel 64 
architecture and 32 bits on processors that do not. 


Fields whose encodings use value 1 are specially treated to allow 32-bit software access to all 64 bits of the 
field. Such access is allowed by defining, for each such field, an encoding that allows direct access to the high 
32 bits of the field. See below. 


Field type. Bits 11:10 encode the type of VMCS field: control, guest-state, host-state, or VM-exit information. 
(The last category also includes the VM-instruction error field. ) 


Index. Bits 9:1 distinguish components with the same field width and type. 


Access type. Bit 0 must be 0 for all fields except for 64-bit fields (those with field-width 1; see above). A 
VMREAD or VMWRITE using an encoding with this bit cleared to 0 accesses the entire field. For a 64-bit field 
with field-width 1, a VMREAD or VMWRITE using an encoding with this bit set to 1 accesses only the high 32 bits 
of the field. 


Appendix B gives the encodings of all fields in the VMCS. 


The following describes the operation of VMREAD and VMWRITE based on processor mode, VMCS-field width, and 


access type: 


16-bit fields: 


— AVMREAD returns the value of the field in bits 15:0 of the destination operand; other bits of the destination 
operand are cleared to 0. 


— AVMWRITE writes the value of bits 15:0 of the source operand into the VMCS field; other bits of the source 
operand are not used. 


32-bit fields: 


— AVMREAD returns the value of the field in bits 31:0 of the destination operand; in 64-bit mode, bits 63:32 
of the destination operand are cleared to 0. 


— AVMWRITE writes the value of bits 31:0 of the source operand into the VMCS field; in 64-bit mode, 
bits 63:32 of the source operand are not used. 


64-bit fields and natural-width fields using the full access type outside IA-32e mode. 


— AVMREAD returns the value of bits 31:0 of the field in its destination operand; bits 63:32 of the field are 
ignored. 


— AVMWRITE writes the value of its source operand to bits 31:0 of the field and clears bits 63:32 of the field. 


64-bit fields and natural-width fields using the full access type in 64-bit mode (only on processors that support 
Intel 64 architecture). 


— AVMREAD returns the value of the field in bits 63:0 of the destination operand 
— AVMWRITE writes the value of bits 63:0 of the source operand into the VMCS field. 
64-bit fields using the high access type. 


— AVMREAD returns the value of bits 63:32 of the field in bits 31:0 of the destination operand; in 64-bit 
mode, bits 63:32 of the destination operand are cleared to 0. 


— AVMWRITE writes the value of bits 31:0 of the source operand to bits 63:32 of the field; in 64-bit mode, 
bits 63:32 of the source operand are not used. 


Software seeking to read a 64-bit field outside IA-32e mode can use VMREAD with the full access type (reading 
bits 31:0 of the field) and VMREAD with the high access type (reading bits 63:32 of the field); the order of the two 


VMREAD executions is not important. Software seeking to modify a 64-bit field outside IA-32e mode should first 
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use VMWRITE with the full access type (establishing bits 31:0 of the field while clearing bits 63:32) and then use 
VMWRITE with the high access type (establishing bits 63:32 of the field). 


23.11.3 Initializing a VMCS 


Software should initialize fields in a VMCS (using VMWRITE) before using the VMCS for VM entry. Failure to do so 
may result in unpredictable behavior; for example, a VM entry may fail for unexplained reasons, or a successful 
transition (VM entry or VM exit) may load processor state with unexpected values. 


It is not necessary to initialize fields that the logical processor will not use. (For example, it is not necessary to 
unitize the MSR-bitmap address if the “use MSR bitmaps” VM-execution control is 0.) 


A processor maintains some VMCS information that cannot be modified with the VMWRITE instruction; this 
includes a VMCS's launch state (see Section 23.1). Such information may be stored in the VMCS data portion of a 
VMCS region. Because the format of this information is implementation-specific, there is no way for software to 
know, when it first allocates a region of memory for use as a VMCS region, how the processor will determine this 
information from the contents of the memory region. 


In addition to its other functions, the VMCLEAR instruction initializes any implementation-specific information in 
the VMCS region referenced by its operand. To avoid the uncertainties of implementation-specific behavior, soft- 
ware should execute VMCLEAR on a VMCS region before making the corresponding VMCS active with VMPTRLD for 
the first time. (Figure 23-1 illustrates how execution of VMCLEAR puts a VMCS into a well-defined state. ) 


The following software usage is consistent with these limitations: 
¢ VMCLEAR should be executed for a VMCS before it is used for VM entry for the first time. 


¢ VMLAUNCH should be used for the first VM entry using a VMCS after VMCLEAR has been executed for that 
VMCS. 


¢ VMRESUME should be used for any subsequent VM entry using a VMCS (until the next execution of VMCLEAR 
for the VMCS). 


It is expected that, in general, VMRESUME will have lower latency than VMLAUNCH. Since “migrating” a VMCS from 
one logical processor to another requires use of VMCLEAR (see Section 23.11.1), which sets the launch state of the 
VMCS to “clear”, such migration requires the next VM entry to be performed using VMLAUNCH. Software devel- 
opers can avoid the performance cost of increased VM-entry latency by avoiding unnecessary migration of a VMCS 
from one logical processor to another. 


23.11.4 Software Access to Related Structures 


In addition to data in the VMCS region itself, VMX non-root operation can be controlled by data structures that are 
referenced by pointers in a VMCS (for example, the I/O bitmaps). While the pointers to these data structures are 
parts of the VMCS, the data structures themselves are not. They are not accessible using VMREAD and VMWRITE 
but by ordinary memory writes. 


Software should ensure that each such data structure is modified only when no logical processor with a current 
VMCS that references it is in VMX non-root operation. Doing otherwise may lead to unpredictable behavior 
(including behaviors identified in Section 23.11.1). Exceptions are made for the following data structures (subject 
to detailed discussion in the sections indicated): EPT paging structures and the data structures used to locate SPP 
vectors (Section 27.3.3); the virtual-APIC page (Section 28.1); the posted interrupt descriptor (Section 28.6); and 
the virtualization-exception information area (Section 24.5.7.2). 


23.11.5 VMXON Region 


Before executing VMXON, software allocates a region of memory (called the VMXON region)! that the logical 
processor uses to support VMX operation. The physical address of this region (the VMXON pointer) is provided in 
an operand to VMXON. The VMXON pointer is subject to the limitations that apply to VMCS pointers: 


1. The amount of memory required for the VMXON region is the same as that required for a VMCS region. This size is implementation 
specific and can be determined by consulting the VMX capability MSR IA32_VMX_BASIC (see Appendix A.1). 
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° The VMXON pointer must be 4-KByte aligned (bits 11:0 must be zero). 
* The VMXON pointer must not set any bits beyond the processor’s physical-address width.2/2 


Before executing VMXON, software should write the VMCS revision identifier (see Section 23.2) to the VMXON 
region. (Specifically, it should write the 31-bit VMCS revision identifier to bits 30:0 of the first 4 bytes of the VMXON 
region; bit 31 should be cleared to 0.) It need not initialize the VMXON region in any other way. Software should 
use a separate region for each logical processor and should not access or modify the VMXON region of a logical 
processor between execution of VMXON and VMXOFF on that logical processor. Doing otherwise may lead to unpre- 
dictable behavior (including behaviors identified in Section 23.11.1). 


1. Software can determine a processor's physical-address width by executing CPUID with BOOOO008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


2. If 1A32_VMX_BASIC[48] is read as 1, the VMXON pointer must not set any bits in the range 63:32; see Appendix A.1. 
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In a virtualized environment using VMX, the guest software stack typically runs on a logical processor in VMX non- 
root operation. This mode of operation is similar to that of ordinary processor operation outside of the virtualized 
environment. This chapter describes the differences between VMX non-root operation and ordinary processor oper- 
ation with special attention to causes of VM exits (which bring a logical processor from VMX non-root operation to 
root operation). The differences between VMX non-root operation and ordinary processor operation are described 
in the following sections: 


® Section 24.1, “Instructions That Cause VM Exits” 

° Section 24.2, “Other Causes of VM Exits” 

* Section 24.3, “Changes to Instruction Behavior in VMX Non-Root Operation” 

° Section 24.4, “Other Changes in VMX Non-Root Operation” 

° Section 24.5, “Features Specific to VMX Non-Root Operation” 

® Section 24.6, “Unrestricted Guests” 

Chapter 25, “VM Entries,” describes the data control structures that govern VMX non-root operation. Chapter 25, 
“VM Entries,” describes the operation of VM entries by which the processor transitions from VMX root operation to 


VMX non-root operation. Chapter 24, “VMX Non-Root Operation,” describes the operation of VM exits by which the 
processor transitions from VMX non-root operation to VMX root operation. 


Chapter 27, “VMX Support for Address Translation,” describes two features that support address translation in VMX 
non-root operation. Chapter 28, “APIC Virtualization and Virtual Interrupts,” describes features that support virtu- 
alization of interrupts and the Advanced Programmable Interrupt Controller (APIC) in VMX non-root operation. 


24.1 INSTRUCTIONS THAT CAUSE VM EXITS 


Certain instructions may cause VM exits if executed in VMX non-root operation. Unless otherwise specified, such 
VM exits are “fault-like,” meaning that the instruction causing the VM exit does not execute and no processor state 
is updated by the instruction. Section 26.1 details architectural state in the context of a VM exit. 


Section 24.1.1 defines the prioritization between faults and VM exits for instructions subject to both. Section 
24.1.2 identifies instructions that cause VM exits whenever they are executed in VMX non-root operation (and thus 
can never be executed in VMX non-root operation). Section 24.1.3 identifies instructions that cause VM exits 
depending on the settings of certain VM-execution control fields (see Section 23.6). 


24.1.1. Relative Priority of Faults and VM Exits 


The following principles describe the ordering between existing faults and VM exits: 


* Certain exceptions have priority over VM exits. These include invalid-opcode exceptions, faults based on 
privilege level,! and general-protection exceptions that are based on checking I/O permission bits in the task- 
state segment (TSS). For example, execution of RDMSR with CPL = 3 generates a general-protection exception 
and not a VM exit. 


¢ Faults incurred while fetching instruction operands have priority over VM exits that are conditioned based on 
the contents of those operands (see LMSW in Section 24.1.3). 


¢ VM exits caused by execution of the INS and OUTS instructions (resulting either because the “unconditional I/O 
exiting” VM-execution control is 1 or because the “use I/O bitmaps control is 1) have priority over the following 
faults: 


1. These include faults generated by attempts to execute, in virtual-8086 mode, privileged instructions that are not recognized in that 
mode. 


2. MOV DRis an exception to this rule; see Section 24.1.3. 
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— A general-protection fault due to the relevant segment (ES for INS; DS for OUTS unless overridden by an 
instruction prefix) being unusable 


— Ageneral-protection fault due to an offset beyond the limit of the relevant segment 
— Analignment-check exception 


¢  Fault-like VM exits have priority over exceptions other than those mentioned above. For example, RDMSR of a 
non-existent MSR with CPL = 0 generates a VM exit and not a general-protection exception. 


When Section 24.1.2 or Section 24.1.3 (below) identify an instruction execution that may lead to a VM exit, it is 
assumed that the instruction does not incur a fault that takes priority over a VM exit. 


24.1.2 Instructions That Cause VM Exits Unconditionally 


The following instructions cause VM exits when they are executed in VMX non-root operation: CPUID, GETSEC,! 
INVD, and XSETBV. This is also true of instructions introduced with VMX, which include: INVEPT, INVVPID, 
VMCALL,? VMCLEAR, VMLAUNCH, VMPTRLD, VMPTRST, VMRESUME, VMXOFF, and VMXON. 


24.1.3 Instructions That Cause VM Exits Conditionally 
Certain instructions cause VM exits in VMX non-root operation depending on the setting of the VM-execution 
controls. The following instructions can cause “fault-like” VM exits based on the conditions described :3 


* CLTS. The CLTS instruction causes a VM exit if the bits in position 3 (corresponding to CRO.TS) are set in both 
the CRO guest/host mask and the CRO read shadow. 


e ENCLS. The ENCLS instruction causes a VM exit if the “enable ENCLS exiting” VM-execution control is 1 and 
one of the following is true: 


— The value of EAX is less than 63 and the corresponding bit in the ENCLS-exiting bitmap is 1 (see Section 
23.6.16). 


— The value of EAX is greater than or equal to 63 and bit 63 in the ENCLS-exiting bitmap is 1. 


¢ ENCLV. The ENCLV instruction causes a VM exit if the “enable ENCLV exiting” VM-execution control is 1 and 
one of the following is true: 


— The value of EAX is less than 63 and the corresponding bit in the ENCLV-exiting bitmap is 1 (see Section 
23.6.17). 


— The value of EAX is greater than or equal to 63 and bit 63 in the ENCLV-exiting bitmap is 1. 
* HLT. The HLT instruction causes a VM exit if the “HLT exiting” VM-execution control is 1. 


e IN, INS/INSB/INSW/INSD, OUT, OUTS/OUTSB/OUTSW/OUTSD. The behavior of each of these instruc- 
tions is determined by the settings of the “unconditional I/O exiting” and “use I/O bitmaps” VM-execution 
controls: 


— If both controls are 0, the instruction executes normally. 


— If the “unconditional I/O exiting” VM-execution control is 1 and the “use I/O bitmaps” VM-execution control 
is 0, the instruction causes a VM exit. 


— If the “use I/O bitmaps” VM-execution control is 1, the instruction causes a VM exit if it attempts to access 
an I/O port corresponding to a bit set to 1 in the appropriate I/O bitmap (see Section 23.6.4). If an I/O 


1. Anexecution of GETSEC in VMX non-root operation causes a VM exit if CR4.SMXE[Bit 14] = 1 regardless of the value of CPL or RAX. 
An execution of GETSEC causes an invalid-opcode exception (HUD) if CR4.SMXE[Bit 14] = 0. 

2. Under the dual-monitor treatment of SMIs and SMM, executions of VMCALL cause SMM VM exits in VMX root operation outside SMM. 
See Section 30.15.2. 

3. Items in this section may refer to secondary processor-based VM-execution controls and tertiary processor-based VM-execution 
controls. If bit 31 of the primary processor-based VM-execution controls is 0, VMX non-root operation functions as if the secondary 
processor-based V/M-execution controls were all 0; similarly, if bit 17 of the primary processor-based VM-execution controls is 0, 
V/MX non-root operation functions as if the tertiary processor-based VM-execution controls were all 0. See Section 23.6.2. 
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operation “wraps around” the 16-bit I/O-port space (accesses ports FFFFH and 0000H), the I/O instruction 
causes a VM exit (the “unconditional I/O exiting” VM-execution control is ignored if the “use I/O bitmaps” 
VM-execution control is 1). 
See Section 24.1.1 for information regarding the priority of VM exits relative to faults that may be caused by 
the INS and OUTS instructions. 


INVLPG. The INVLPG instruction causes a VM exit if the “INVLPG exiting” VM-execution control is 1. 


INVPCID. The INVPCID instruction causes a VM exit if the “INVLPG exiting” and “enable INVPCID” 
VM-execution controls are both 1. 

LGDT, LIDT, LLDT, LTR, SGDT, SIDT, SLDT, STR. These instructions cause VM exits if the “descriptor-table 
exiting” VM-execution control is 1. 

LMSW. In general, the LMSW instruction causes a VM exit if it would write, for any bit set in the low 4 bits of 


the CRO guest/host mask, a value different than the corresponding bit in the CRO read shadow. LMSW never 
clears bit 0 of CRO (CRO.PE); thus, LMSW causes a VM exit if either of the following are true: 


— The bits in position 0 (corresponding to CRO.PE) are set in both the CRO guest/host mask and the source 
operand, and the bit in position 0 is clear in the CRO read shadow. 


— For any bit position in the range 3:1, the bit in that position is set in the CRO guest/host mask and the 
values of the corresponding bits in the source operand and the CRO read shadow differ. 

LOADIWKEY. The LOADIWKEY instruction causes a VM exit if the “LOADIWKEY exiting” VM-execution control 

is 1. 

MONITOR. The MONITOR instruction causes a VM exit if the “MONITOR exiting” VM-execution control is 1. 

MOV from CR3. The MOV from CR3 instruction causes a VM exit if the “CR3-store exiting” VM-execution 

control is 1. The first processors to support the virtual-machine extensions supported only the 1-setting of this 

control. 


MOV from CR8. The MOV from CR8 instruction causes a VM exit if the “CR8-store exiting” VM-execution 
control is 1. 


MOV to CRO. The MOV to CRO instruction causes a VM exit unless the value of its source operand matches, for 
the position of each bit set in the CRO guest/host mask, the corresponding bit in the CRO read shadow. (If every 
bit is clear in the CRO guest/host mask, MOV to CRO cannot cause a VM exit.) 

MOV to CR3. The MOV to CR3 instruction causes a VM exit unless the “CR3-load exiting” VM-execution control 
is O or the value of its source operand is equal to one of the CR3-target values specified in the VMCS. Only the 
first n CR3-target values are considered, where n is the CR3-target count. If the “CR3-load exiting” VM- 
execution control is 1 and the CR3-target count is 0, MOV to CR3 always causes a VM exit. 


The first processors to support the virtual-machine extensions supported only the 1-setting of the “CR3-load 
exiting” VM-execution control. These processors always consult the CR3-target controls to determine whether 
an execution of MOV to CR3 causes a VM exit. 


MOV to CR4. The MOV to CR4 instruction causes a VM exit unless the value of its source operand matches, for 
the position of each bit set in the CR4 guest/host mask, the corresponding bit in the CR4 read shadow. 


MOV to CR8. The MOV to CR8 instruction causes a VM exit if the “CR8-load exiting” VM-execution control is 1. 


MOV DR. The MOV DR instruction causes a VM exit if the “MOV-DR exiting” VM-execution control is 1. Such 
VM exits represent an exception to the principles identified in Section 24.1.1 in that they take priority over the 
following: general-protection exceptions based on privilege level; and invalid-opcode exceptions that occur 
because CR4.DE=1 and the instruction specified access to DR4 or DR5. 


MWAIT. The MWAIT instruction causes a VM exit if the “MWAIT exiting” VM-execution control is 1. If this 
control is 0, the behavior of the MWAIT instruction may be modified (see Section 24.3). 


PAUSE. The behavior of each of this instruction depends on CPL and the settings of the “PAUSE exiting” and 
“PAUSE-loop exiting” VM-execution controls: 


— CPL=0. 


¢ If the “PAUSE exiting” and “PAUSE-loop exiting” VM-execution controls are both 0, the PAUSE 
instruction executes normally. 
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¢ If the “PAUSE exiting” VM-execution control is 1, the PAUSE instruction causes a VM exit (the “PAUSE- 
loop exiting” VM-execution control is ignored if CPL = 0 and the “PAUSE exiting” VM-execution control 
is 1). 

¢ If the “PAUSE exiting” VM-execution control is 0 and the “PAUSE-loop exiting” VM-execution control is 
1, the following treatment applies. 


The processor determines the amount of time between this execution of PAUSE and the previous 
execution of PAUSE at CPL 0. If this amount of time exceeds the value of the VM-execution control field 
PLE_Gap, the processor considers this execution to be the first execution of PAUSE in a loop. (It also 
does so for the first execution of PAUSE at CPL 0 after VM entry.) 


Otherwise, the processor determines the amount of time since the most recent execution of PAUSE that 
was considered to be the first in a loop. If this amount of time exceeds the value of the VM-execution 
control field PLE_Window, a VM exit occurs. 


For purposes of these computations, time is measured based on a counter that runs at the same rate as 
the timestamp counter (TSC). 


— CPL>O. 
¢ If the “PAUSE exiting” VM-execution control is 0, the PAUSE instruction executes normally. 
¢ If the “PAUSE exiting” VM-execution control is 1, the PAUSE instruction causes a VM exit. 
The “PAUSE-loop exiting” VM-execution control is ignored if CPL > 0. 
¢ RDMSR. The RDMSR instruction causes a VM exit if any of the following are true: 
— The “use MSR bitmaps” VM-execution control is 0. 
— The value of ECX is not in the ranges OOOO0000H - OOO01FFFH and COOOOO0OH - COOO1FFFH. 


— The value of ECX is in the range OOOO0000H — OOOO1FFFH and bit rn in read bitmap for low MSRs is 1, where 
nis the value of ECX. 


— The value of ECX is in the range COOOO000H - COOO1FFFH and bit rn in read bitmap for high MSRs is 1, 
where n is the value of ECX & OOOO1FFFH. 


See Section 23.6.9 for details regarding how these bitmaps are identified. 
* RDPMC. The RDPMC instruction causes a VM exit if the “RDPMC exiting” VM-execution control is 1. 
¢ RDRAND. The RDRAND instruction causes a VM exit if the “RDRAND exiting” VM-execution control is 1. 
¢ RDSEED. The RDSEED instruction causes a VM exit if the “RDSEED exiting” VM-execution control is 1. 
° RDTSC. The RDTSC instruction causes a VM exit if the “RDTSC exiting” VM-execution control is 1. 


° RDTSCP. The RDTSCP instruction causes a VM exit if the “RDTSC exiting” and “enable RDTSCP” VM-execution 
controls are both 1. 


* RSM. The RSM instruction causes a VM exit if executed in system-management mode (SMM).! 


¢ TPAUSE. The TPAUSE instruction causes a VM exit if the “RDTSC exiting” and “enable user wait and pause” 
VM-execution controls are both 1. 


¢ UMWATT. The UMWAIT instruction causes a VM exit if the “RDTSC exiting” and “enable user wait and pause” 
VM-execution controls are both 1. 


© VMREAD. The VMREAD instruction causes a VM exit if any of the following are true: 
— The“VMCS shadowing” VM-execution control is 0. 
— Bits 63:15 (bits 31:15 outside 64-bit mode) of the register source operand are not all 0. 


— Bit n in VMREAD bitmap is 1, where n is the value of bits 14:0 of the register source operand. See Section 
23.6.15 for details regarding how the VMREAD bitmap is identified. 


1. Execution of the RSM instruction outside SMM causes an invalid-opcode exception regardless of whether the processor is in VMX 
operation. It also does so in VMX root operation in SMM; see Section 30.15.3. 
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If the VMREAD instruction does not cause a VM exit, it reads from the VMCS referenced by the VMCS link 
pointer. See Chapter 29, “VMREAD—Read Field from Virtual-Machine Control Structure” for details of the 
operation of the VMREAD instruction. 


VMWRITE. The VMWRITE instruction causes a VM exit if any of the following are true: 
— The“VMCS shadowing” VM-execution control is 0. 
— Bits 63:15 (bits 31:15 outside 64-bit mode) of the register source operand are not all 0. 


— Bit nin VMWRITE bitmap is 1, where rn is the value of bits 14:0 of the register source operand. See Section 
23.6.15 for details regarding how the VMWRITE bitmap is identified. 


If the VMWRITE instruction does not cause a VM exit, it writes to the VMCS referenced by the VMCS link 
pointer. See Chapter 29, “VMWRITE—Write Field to Virtual-Machine Control Structure” for details of the 
operation of the VMWRITE instruction. 


WBINVD. The WBINVD instruction causes a VM exit if the “WBINVD exiting” VM-execution control is 1. 
WBNOINVD. The WBNOINVD instruction causes a VM exit if the “WBINVD exiting” VM-execution control is 1. 
WRMSR. The WRMSR instruction causes a VM exit if any of the following are true: 

— The “use MSR bitmaps” VM-execution control is 0. 

— The value of ECX is not in the ranges OOOO0000H - OOOO1FFFH and COOOOQO0OH - COOO1FFFH. 


— The value of ECX is in the range OOOO0000H - OOOO1FFFH and bit rn in write bitmap for low MSRs is 1, 
where n is the value of ECX. 


— The value of ECX is in the range COOO0000H - COOOLFFFH and bit rn in write bitmap for high MSRs is 1, 
where n is the value of ECX & OOOO1FFFH. 


See Section 23.6.9 for details regarding how these bitmaps are identified. 


XRSTORS. The XRSTORS instruction causes a VM exit if the “enable XSAVES/XRSTORS” VM-execution control 
is Land any bit is set in the logical-AND of the following three values: EDX:EAX, the IA32_XSS MSR, and the 
XSS-exiting bitmap (see Section 23.6.20). 


XSAVES. The XSAVES instruction causes a VM exit if the “enable XSAVES/XRSTORS” VM-execution control is 
1 and any bit is set in the logical-AND of the following three values: EDX:EAX, the IA32_XSS MSR, and the XSS- 
exiting bitmap (see Section 23.6.20). 


24.2 OTHER CAUSES OF VM EXITS 


In addition to VM exits caused by instruction execution, the following events can cause VM exits: 


Exceptions. Exceptions (faults, traps, and aborts) cause VM exits based on the exception bitmap (see Section 
23.6.3). If an exception occurs, its vector (in the range 0-31) is used to select a bit in the exception bitmap. If 
the bit is 1, a VM exit occurs; if the bit is 0, the exception is delivered normally through the guest IDT. This use 
of the exception bitmap applies also to exceptions generated by the instructions INT1, INT3, INTO, BOUND, 
UDO, UD1, and UD2.? 


Page faults (exceptions with vector 14) are specially treated. When a page fault occurs, a processor consults 
(1) bit 14 of the exception bitmap; (2) the error code produced with the page fault [PFEC]; (3) the page-fault 
error-code mask field [PFEC_MASK]; and (4) the page-fault error-code match field [PFEC_MATCH]. It checks if 
PFEC & PFEC_MASK = PFEC_MATCH. If there is equality, the specification of bit 14 in the exception bitmap is 
followed (for example, a VM exit occurs if that bit is set). If there is inequality, the meaning of that bit is 
reversed (for example, a VM exit occurs if that bit is clear). 


Thus, if software desires VM exits on all page faults, it can set bit 14 in the exception bitmap to 1 and set the 
page-fault error-code mask and match fields each to QOOOOOOOH. If software desires VM exits on no page 
faults, it can set bit 14 in the exception bitmap to 1, the page-fault error-code mask field to QOOOO000H, and 
the page-fault error-code match field to FFFFFFFFH. 


1. 


INT1 and INT3 refer to the instructions with opcodes F1 and CC, respectively, and not to INT n with value 1 or 3 for n. 
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Triple fault. A VM exit occurs if the logical processor encounters an exception while attempting to call the 
double-fault handler and that exception itself does not cause a VM exit due to the exception bitmap. This 
applies to the case in which the double-fault exception was generated within VMX non-root operation, the case 
in which the double-fault exception was generated during event injection by VM entry, and to the case in which 
VM entry is injecting a double-fault exception. 


External interrupts. An external interrupt causes a VM exit if the “external-interrupt exiting” VM-execution 
control is 1. (See Section 24.6 for an exception.) Otherwise, the interrupt is delivered normally through the 
IDT. (If a logical processor is in the shutdown state or the wait-for-SIPI state, external interrupts are blocked. 
The interrupt is not delivered through the IDT and no VM exit occurs.) 


Non-maskable interrupts (NMIs). An NMI causes a VM exit if the “NMI exiting” VM-execution control is 1. 
Otherwise, it is delivered using descriptor 2 of the IDT. (If a logical processor is in the wait-for-SIPI state, NMIs 
are blocked. The NMI is not delivered through the IDT and no VM exit occurs.) 


INIT signals. INIT signals cause VM exits. A logical processor performs none of the operations normally 
associated with these events. Such exits do not modify register state or clear pending events as they would 
outside of VMX operation. (If a logical processor is in the wait-for-SIPI state, INIT signals are blocked. They do 
not cause VM exits in this case.) 


Start-up IPIs (SIPIs). SIPIs cause VM exits. If a logical processor is not in the wait-for-SIPI activity state 
when a SIPI arrives, no VM exit occurs and the SIPI is discarded. VM exits due to SIPIs do not perform any of 
the normal operations associated with those events: they do not modify register state as they would outside of 
VMX operation. (If a logical processor is not in the wait-for-SIPI state, SIPIs are blocked. They do not cause 
VM exits in this case.) 


Task switches. Task switches are not allowed in VMX non-root operation. Any attempt to effect a task switch 
in VMX non-root operation causes a VM exit. See Section 24.4.2. 


System-management interrupts (SMIs). If the logical processor is using the dual-monitor treatment of 
SMIs and system-management mode (SMM), SMIs cause SMM VM exits. See Section 30.15.2.! 


VMX-preemption timer. A VM exit occurs when the timer counts down to zero. See Section 24.5.1 for details 
of operation of the VMX-preemption timer. 


Debug-trap exceptions and higher priority events take priority over VM exits caused by the VMX-preemption 
timer. VM exits caused by the VMX-preemption timer take priority over VM exits caused by the “NMI-window 
exiting” VM-execution control and lower priority events. 


These VM exits wake a logical processor from the same inactive states as would a non-maskable interrupt. 
Specifically, they wake a logical processor from the shutdown state and from the states entered using the HLT 
and MWAIT instructions. These VM exits do not occur if the logical processor is in the wait-for-SIPI state. 


In addition, there are controls that cause VM exits based on the readiness of guest software to receive interrupts: 


If the “interrupt-window exiting” VM-execution control is 1,a VM exit occurs before execution of any instruction 
if RFLAGS.IF = 1 and there is no blocking of events by STI or by MOV SS (see Table 23-3). Such a VM exit 
occurs immediately after VM entry if the above conditions are true (see Section 25.7.5). 


Non-maskable interrupts (NMIs) and higher priority events take priority over VM exits caused by this control. 
VM exits caused by this control take priority over external interrupts and lower priority events. 


These VM exits wake a logical processor from the same inactive states as would an external interrupt. Specifi- 
cally, they wake a logical processor from the states entered using the HLT and MWAIT instructions. These 
VM exits do not occur if the logical processor is in the shutdown state or the wait-for-SIPI state. 


If the “NMI-window exiting” VM-execution control is 1, a VM exit occurs before execution of any instruction if 
there is no virtual-NMI blocking and there is no blocking of events by MOV SS and no blocking of events by STI 
(see Table 23-3). Such a VM exit occurs immediately after VM entry if the above conditions are true (see 
Section 25.7.6). 


VM exits caused by the VMX-preemption timer and higher priority events take priority over VM exits caused by 
this control. VM exits caused by this control take priority over non-maskable interrupts (NMIs) and lower 
priority events. 


1. 


Under the dual-monitor treatment of SMIs and SMM, SMIs also cause SMM VM exits if they occur in VMX root operation outside SMM. 
If the processor is using the default treatment of SMIs and SMM, SMls are delivered as described in Section 30.14.1. 
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These VM exits wake a logical processor from the same inactive states as would an NMI. Specifically, they wake 


a logical processor from the shutdown state and from the states entered using the HLT and MWAIT instructions. 


These VM exits do not occur if the logical processor is in the wait-for-SIPI state. 


24.3. CHANGES TO INSTRUCTION BEHAVIOR IN VMX NON-ROOT OPERATION 


The behavior of some instructions is changed in VMX non-root operation. Some of these changes are determined 
by the settings of certain VM-execution control fields. The following items detail such changes: 


CLTS. Behavior of the CLTS instruction is determined by the bits in position 3 (corresponding to CRO.TS) in the 
CRO guest/host mask and the CRO read shadow: 


— If bit 3 in the CRO guest/host mask is 0, CLTS clears CRO.TS normally (the value of bit 3 in the CRO read 
shadow is irrelevant in this case), unless CRO.TS is fixed to 1 in VMX operation (see Section 22.8), in which 
case CLTS causes a general-protection exception. 


— If bit 3 in the CRO guest/host mask is 1 and bit 3 in the CRO read shadow is 0, CLTS completes but does not 
change the contents of CRO.TS. 


— Ifthe bits in position 3 in the CRO guest/host mask and the CRO read shadow are both 1, CLTS causes a 
VM exit. 


INVPCID. Behavior of the INVPCID instruction is determined first by the setting of the “enable INVPCID” 
VM-execution control: 


— Ifthe “enable INVPCID” VM-execution control is 0, INVPCID causes an invalid-opcode exception (#UD). 
This exception takes priority over any other exception the instruction may incur. 


— Ifthe “enable INVPCID” VM-execution control is 1, treatment is based on the setting of the “INVLPG 
exiting” VM-execution control: 


¢ Ifthe “INVLPG exiting” VM-execution control is 0, INVPCID operates normally. 
¢ If the “INVLPG exiting” VM-execution control is 1, INVPCID causes a VM exit. 


IRET. Behavior of IRET with regard to NMI blocking (see Table 23-3) is determined by the settings of the “NMI 
exiting” and “virtual NMIs” VM-execution controls: 


— Ifthe “NMI exiting” VM-execution control is 0, IRET operates normally and unblocks NMIs. (If the “NMI 
exiting” VM-execution control is 0, the “virtual NMIs” control must be 0; see Section 25.2.1.1.) 


— Ifthe “NMI exiting” VM-execution control is 1, IRET does not affect blocking of NMIs. If, in addition, the 
“virtual NMIs” VM-execution control is 1, the logical processor tracks virtual-NMI blocking. In this case, 
IRET removes any virtual-NMI blocking. 


The unblocking of NMIs or virtual NMIs specified above occurs even if IRET causes a fault. 


LMSW. Outside of VMX non-root operation, LMSW loads its source operand into CRO[3:0], but it does not clear 
CRO.PE if that bit is set. In VMX non-root operation, an execution of LMSW that does not cause a VM exit (see 
Section 24.1.3) leaves unmodified any bit in CRO[3:0] corresponding to a bit set in the CRO guest/host mask. 
An attempt to set any other bit in CRO[3:0] to a value not supported in VMX operation (see Section 22.8) 
causes a general-protection exception. Attempts to clear CRO.PE are ignored without fault. 


MOV from CRO. The behavior of MOV from CRO is determined by the CRO guest/host mask and the CRO read 
shadow. For each position corresponding to a bit clear in the CRO guest/host mask, the destination operand is 
loaded with the value of the corresponding bit in CRO. For each position corresponding to a bit set in the CRO 
guest/host mask, the destination operand is loaded with the value of the corresponding bit in the CRO read 
shadow. Thus, if every bit is cleared in the CRO guest/host mask, MOV from CRO reads normally from CRO; if 
every bit is set in the CRO guest/host mask, MOV from CRO returns the value of the CRO read shadow. 


Items in this section may refer to secondary processor-based VM-execution controls and tertiary processor-based VM-execution 
controls. If bit 31 of the primary processor-based VM-execution controls is 0, VMX non-root operation functions as if the secondary 
processor-based VM-execution controls were all 0; similarly, if bit 17 of the primary processor-based VM-execution controls is 0, 
V/MX non-root operation functions as if the tertiary processor-based VM-execution controls were all 0. See Section 23.6.2. 
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Depending on the contents of the CRO guest/host mask and the CRO read shadow, bits may be set in the 
destination that would never be set when reading directly from CRO. 


¢ MOV from CR3. If the “enable EPT” VM-execution control is 1 and an execution of MOV from CR3 does not 
cause a VM exit (See Section 24.1.3), the value loaded from CR3 is a guest-physical address; see Section 
27.2.1. 


¢ MOV from CR4. The behavior of MOV from CR4 is determined by the CR4 guest/host mask and the CR4 read 
shadow. For each position corresponding to a bit clear in the CR4 guest/host mask, the destination operand is 
loaded with the value of the corresponding bit in CR4. For each position corresponding to a bit set in the CR4 
guest/host mask, the destination operand is loaded with the value of the corresponding bit in the CR4 read 
shadow. Thus, if every bit is cleared in the CR4 guest/host mask, MOV from CR4 reads normally from CR4; if 
every bit is set in the CR4 guest/host mask, MOV from CR4 returns the value of the CR4 read shadow. 


Depending on the contents of the CR4 guest/host mask and the CR4 read shadow, bits may be set in the 
destination that would never be set when reading directly from CR4. 


¢ MOV from CR8. If the MOV from CR8 instruction does not cause a VM exit (see Section 24.1.3), its behavior is 
modified if the “use TPR shadow” VM-execution control is 1; see Section 28.3. 


¢ MOV to CRO. An execution of MOV to CRO that does not cause a VM exit (see Section 24.1.3) leaves 
unmodified any bit in CRO corresponding to a bit set in the CRO guest/host mask. Treatment of attempts to 
modify other bits in CRO depends on the setting of the “unrestricted guest” VM-execution control: 


— Ifthe control is 0, MOV to CRO causes a general-protection exception if it attempts to set any bit in CRO to 
a value not supported in VMX operation (see Section 22.8). 


— Ifthe control is 1, MOV to CRO causes a general-protection exception if it attempts to set any bit in CRO 
other than bit 0 (PE) or bit 31 (PG) to a value not supported in VMX operation. It remains the case, 
however, that MOV to CRO causes a general-protection exception if it would result in CRO.PE = 0 and 
CRO.PG = 1 or if it would result in CRO.PG = 1, CR4.PAE = 0, and IA32_EFER.LME = 1. 


° MOV to CR3. If the “enable EPT” VM-execution control is 1 and an execution of MOV to CR3 does not cause a 
VM exit (see Section 24.1.3), the value loaded into CR3 is treated as a guest-physical address; see Section 
27.2.1. 


— If PAE paging is not being used, the instruction does not use the guest-physical address to access memory 
and it does not cause it to be translated through EPT.! 


— If PAE paging is being used, the instruction translates the guest-physical address through EPT and uses the 
result to load the four (4) page-directory-pointer-table entries (PDPTEs). The instruction does not use the 
guest-physical addresses the PDPTEs to access memory and it does not cause them to be translated 
through EPT. 


¢ MOV to CR4. An execution of MOV to CR4 that does not cause a VM exit (see Section 24.1.3) leaves 
unmodified any bit in CR4 corresponding to a bit set in the CR4 guest/host mask. Such an execution causes a 
general-protection exception if it attempts to set any bit in CR4 (not corresponding to a bit set in the CR4 
guest/host mask) to a value not supported in VMX operation (see Section 22.8). 

° MOV to CR8. If the MOV to CR8 instruction does not cause a VM exit (see Section 24.1.3), its behavior is 
modified if the “use TPR shadow” VM-execution control is 1; see Section 28.3. 


° MWAIT. Behavior of the MWAIT instruction (which always causes an invalid-opcode exception—#UD—if 
CPL > 0) is determined by the setting of the “MWAIT exiting” VM-execution control: 


— Ifthe “MWAIT exiting” VM-execution control is 1, MWAIT causes a VM exit. 


— Ifthe “MWAIT exiting” VM-execution control is 0, MWAIT operates normally if one of the following are true: 
(1) ECX[0] is 0; (2) RFLAGS.IF = 1; or both of the following are true: (a) the “interrupt-window exiting” VM- 
execution control is 0; and (b) the logical processor has not recognized a pending virtual interrupt (see 
Section 29.2.1). 


— If the “MWAIT exiting” VM-execution control is 0, ECX[0] = 1, and RFLAGS.IF = 0, MWAIT does not cause 
the processor to enter an implementation-dependent optimized state if either the “interrupt-window 


1. A logical processor uses PAE paging if CRO.PG = 1, CR4.PAE = 1 and IA32_EFER.LMA = O. See Section 4.4 in the Intel® 64 and IA-32 
Architectures Software Developer's Manual, Volume 3A. 
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exiting” VM-execution control is 1 or the logical processor has recognized a pending virtual interrupt; 
instead, control passes to the instruction following the MWAIT instruction. 


RDMSR. Section 24.1.3 identifies when executions of the RDMSR instruction cause VM exits. If such an 
execution causes neither a fault due to CPL > 0 nor a VM exit, the instruction’s behavior may be modified for 
certain values of ECX: 


— If ECX contains 10H (indicating the IA32_TIME_STAMP_COUNTER MSR), the value returned by the 
instruction is determined by the setting of the “use TSC offsetting” VM-execution control: 


¢ If the control is 0, RDMSR operates normally, loading EAX:EDX with the value of the 
IA32_TIME_STAMP_COUNTER MSR. 


¢ Ifthe control is 1, the value returned is determined by the setting of the “use TSC scaling” 
VM-execution control: 


— If the control is 0, RDMSR loads EAX:EDX with the sum of the value of the 
IA32_TIME_STAMP_COUNTER MSR and the value of the TSC offset. 


— If the control is 1, RDMSR first computes the product of the value of the 
IA32_TIME_STAMP_COUNTER MSR and the value of the TSC multiplier. It then shifts the value of 
the product right 48 bits and loads EAX:EDX with the sum of that shifted value and the value of the 
TSC offset. 


The 1-setting of the “use TSC-offsetting” VM-execution control does not affect executions of RDMSR if ECX 
contains 6EOH (indicating the IA32_TSC_DEADLINE MSR). Such executions return the APIC-timer deadline 
relative to the actual timestamp counter without regard to the TSC offset. 


— If ECXis in the range 800H-8FFH (indicating an APIC MSR), instruction behavior may be modified if the 
“virtualize x2APIC mode” VM-execution control is 1; see Section 28.5. 


RDPID. Behavior of the RDPID instruction is determined first by the setting of the “enable RDTSCP” 
VM-execution control: 


— Ifthe “enable RDTSCP” VM-execution control is 0, RDPID causes an invalid-opcode exception (#UD). 
— Ifthe “enable RDTSCP” VM-execution control is 1, RDPID operates normally. 


RDTSC. Behavior of the RDTSC instruction is determined by the settings of the “RDTSC exiting” and “use TSC 
offsetting” VM-execution controls: 


— If both controls are 0, RDTSC operates normally. 


— Ifthe“RDTSC exiting” VM-execution control is 0 and the “use TSC offsetting” VM-execution control is 1, the 
value returned is determined by the setting of the “use TSC scaling” VM-execution control: 


¢ Ifthe control is 0, RDTSC loads EAX:EDX with the sum of the value of the IA32_TIME_STAMP_COUNTER 
MSR and the value of the TSC offset. 


¢ If the control is 1, RDTSC first computes the product of the value of the IA32_TIME_STAMP_COUNTER 
MSR and the value of the TSC multiplier. It then shifts the value of the product right 48 bits and loads 
EAX:EDX with the sum of that shifted value and the value of the TSC offset. 


— Ifthe “RDTSC exiting” VM-execution control is 1, RDTSC causes a VM exit. 


RDTSCP. Behavior of the RDTSCP instruction is determined first by the setting of the “enable RDTSCP” 
VM-execution control: 


— Ifthe“enable RDTSCP” VM-execution control is 0, RDTSCP causes an invalid-opcode exception (#UD). This 
exception takes priority over any other exception the instruction may incur. 


— Ifthe “enable RDTSCP” VM-execution control is 1, treatment is based on the settings of the “RDTSC exiting” 
and “use TSC offsetting” VM-execution controls: 


¢ If both controls are 0, RDTSCP operates normally. 


¢ Ifthe“RDTSC exiting” VM-execution control is 0 and the “use TSC offsetting” VM-execution control is 1, 
the value returned is determined by the setting of the “use TSC scaling” VM-execution control: 


— If the control is 0, RDTSCP loads EAX:EDX with the sum of the value of the 
IA32_TIME_STAMP_COUNTER MSR and the value of the TSC offset. 
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— If the control is 1, RDTSCP first computes the product of the value of the 
IA32_TIME_STAMP_COUNTER MSR and the value of the TSC multiplier. It then shifts the value of 
the product right 48 bits and loads EAX:EDX with the sum of that shifted value and the value of the 
TSC offset. 


In either case, RDTSCP also loads ECX with the value of bits 31:0 of the IA32_TSC_AUX MSR. 
¢ If the “RDTSC exiting” VM-execution control is 1, RDTSCP causes a VM exit. 


¢ SMSW. The behavior of SMSW is determined by the CRO guest/host mask and the CRO read shadow. For each 
position corresponding to a bit clear in the CRO guest/host mask, the destination operand is loaded with the 
value of the corresponding bit in CRO. For each position corresponding to a bit set in the CRO guest/host mask, 
the destination operand is loaded with the value of the corresponding bit in the CRO read shadow. Thus, if every 
bit is cleared in the CRO guest/host mask, SMSW reads normally from CRO; if every bit is set in the CRO 
guest/host mask, SMSW returns the value of the CRO read shadow. 


Note the following: (1) for any memory destination or for a 16-bit register destination, only the low 16 bits of 
the CRO guest/host mask and the CRO read shadow are used (bits 63:16 of a register destination are left 
unchanged); (2) for a 32-bit register destination, only the low 32 bits of the CRO guest/host mask and the CRO 
read shadow are used (bits 63:32 of the destination are cleared); and (3) depending on the contents of the 
CRO guest/host mask and the CRO read shadow, bits may be set in the destination that would never be set 
when reading directly from CRO. 


* TPAUSE. Behavior of the TPAUSE instruction is determined first by the setting of the “enable user wait and 
pause” VM-execution control: 


— If the “enable user wait and pause” VM-execution control is 0, TPAUSE causes an invalid-opcode exception 
(#UD). This exception takes priority over any exception the instruction may incur. 


— Ifthe “enable user wait and pause” VM-execution control is 1, treatment is based on the setting of the 
“RDTSC exiting” VM-execution control: 


¢ Ifthe “RDTSC exiting” VM-execution control is 0, the instruction delays for an amount of time called 
here the physical delay. The physical delay is first computed by determining the virtual delay (the 
time to delay relative to the guest’s timestamp counter). 


If IA32_UMWAIT_CONTROL[31:2] is zero, the virtual delay is the value in EDX:EAX minus the value 
that RDTSC would return (see above); if IA32_UMWAIT_CONTROL[31:2] is not zero, the virtual delay 
is the minimum of that difference and AND(IA32_UMWAIT_CONTROL,FFFFFFFCH). 


The physical delay depends upon the settings of the “use TSC offsetting” and “use TSC scaling” 
VM-execution controls: 


— If either control is 0, the physical delay is the virtual delay. 


— If both controls are 1, the virtual delay is multiplied by 248 (using a shift) to produce a 128-bit 
integer. That product is then divided by the TSC multiplier to produce a 64-bit integer. The physical 
delay is that quotient. 


¢ If the “RDTSC exiting” VM-execution control is 1, TPAUSE causes a VM exit. 


¢ UMONITOR. Behavior of the UMONITOR instruction is determined by the setting of the “enable user wait and 
pause” VM-execution control: 


— Ifthe “enable user wait and pause” VM-execution control is 0, UMONITOR causes an invalid-opcode 
exception (#UD). This exception takes priority over any exception the instruction may incur. 


— Ifthe “enable user wait and pause” VM-execution control is 1, UMONITOR operates normally. 


° UMWATT. Behavior of the UMWAIT instruction is determined first by the setting of the “enable user wait and 
pause” VM-execution control: 


— Ifthe “enable user wait and pause” VM-execution control is 0, UMWAIT causes an invalid-opcode exception 
(#UD). This exception takes priority over any exception the instruction may incur. 


— Ifthe “enable user wait and pause” VM-execution control is 1, treatment is based on the setting of the 
“RDTSC exiting” VM-execution control: 
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¢ Ifthe “RDTSC exiting” VM-execution control is 0, and if the instruction causes a delay, the amount of 
time delayed is called here the physical delay. The physical delay is first computed by determining the 
virtual delay (the time to delay relative to the guest’s timestamp counter). 


If IA32_UMWAIT_CONTROL[31:2] is zero, the virtual delay is the value in EDX:EAX minus the value 
that RDTSC would return (see above); if IA32_UMWAIT_CONTROL[31:2] is not zero, the virtual delay 
is the minimum of that difference and AND(IA32_UMWAIT_CONTROL,FFFFFFFCH). 


The physical delay depends upon the settings of the “use TSC offsetting” and “use TSC scaling” 
VM-execution controls: 


— If either control is 0, the physical delay is the virtual delay. 


— If both controls are 1, the virtual delay is multiplied by 248 (using a shift) to produce a 128-bit 
integer. That product is then divided by the TSC multiplier to produce a 64-bit integer. The physical 
delay is that quotient. 


¢ Ifthe “RDTSC exiting” VM-execution control is 1, UMWAIT causes a VM exit. 


WRMSR. Section 24.1.3 identifies when executions of the WRMSR instruction cause VM exits. If such an 
execution neither a fault due to CPL > 0 nor a VM exit, the instruction’s behavior may be modified for certain 
values of ECX: 


— If ECX contains 79H (indicating IA32_BIOS_UPDT_TRIG MSR), no microcode update is loaded, and control 
passes to the next instruction. This implies that microcode updates cannot be loaded in VMX non-root 
operation. 


— On processors that support Intel PT but which do not allow it to be used in VMX operation, if ECX contains 
570H (indicating the IA32_RTIT_CTL MSR), the instruction causes a general-protection exception.! 


— If ECX contains 808H (indicating the TPR MSR), 80BH (the EOI MSR), or 83FH (self-IPI MSR), instruction 
behavior may modified if the “virtualize x2APIC mode” VM-execution control is 1; see Section 28.5. 


XRSTORS. Behavior of the XRSTORS instruction is determined first by the setting of the “enable 
XSAVES/XRSTORS” VM-execution control: 


— Ifthe “enable XSAVES/XRSTORS” VM-execution control is 0, XRSTORS causes an invalid-opcode exception 
(#UD). 


— Ifthe “enable XSAVES/XRSTORS” VM-execution control is 1, treatment is based on the value of the XSS- 
exiting bitmap (see Section 23.6.20): 


¢ XRSTORS causes a VM exit if any bit is set in the logical-AND of the following three values: EDX:EAX, 
the IA32_XSS MSR, and the XSS-exiting bitmap. 


* Otherwise, XRSTORS operates normally. 


XSAVES. Behavior of the XSAVES instruction is determined first by the setting of the “enable 
XSAVES/XRSTORS” VM-execution control: 


— If the “enable XSAVES/XRSTORS” VM-execution control is 0, XSAVES causes an invalid-opcode exception 
(#UD). 


— If the “enable XSAVES/XRSTORS” VM-execution control is 1, treatment is based on the value of the XSS- 
exiting bitmap (see Section 23.6.20): 


¢ XSAVES causes a VM exit if any bit is set in the logical-AND of the following three values: EDX:EAX, the 
IA32_XSS MSR, and the XSS-exiting bitmap. 


¢ Otherwise, XSAVES operates normally. 


24.4 OTHER CHANGES IN VMX NON-ROOT OPERATION 


Treatments of event blocking and of task switches differ in VMX non-root operation as described in the following 
sections. 


. Software should read the VMX capability MSR 1[A32_VMX_MISC to determine whether the processor allows Intel PT to be used in 
VMX operation (see Appendix A.6). 
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24.4.1 Event Blocking 


Event blocking is modified in VMX non-root operation as follows: 


° If the “external-interrupt exiting” VM-execution control is 1, RFLAGS.IF does not control the blocking of 
external interrupts. In this case, an external interrupt that is not blocked for other reasons causes a VM exit 
(even if RFLAGS.IF = 0). 


e If the “external-interrupt exiting” VM-execution control is 1, external interrupts may or may not be blocked by 
STI or by MOV SS (behavior is implementation-specific). 


e If the “NMI exiting” VM-execution control is 1, non-maskable interrupts (NMIs) may or may not be blocked by 
STI or by MOV SS (behavior is implementation-specific). 


24.4.2 Treatment of Task Switches 


Task switches are not allowed in VMX non-root operation. Any attempt to effect a task switch in VMX non-root oper- 
ation causes a VM exit. However, the following checks are performed (in the order indicated), possibly resulting in 
a fault, before there is any possibility of a VM exit due to task switch: 


1. Ifa task gate is being used, appropriate checks are made on its P bit and on the proper values of the relevant 
privilege fields. The following cases detail the privilege checks performed: 


a. If CALL, INT n, INT1, INT3, INTO, or JMP accesses a task gate in IA-32e mode, a general-protection 
exception occurs. 


b. If CALL, INT n, INT3, INTO, or JMP accesses a task gate outside IA-32e mode, privilege-levels checks are 
performed on the task gate but, if they pass, privilege levels are not checked on the referenced task-state 
segment (TSS) descriptor. 


If CALL or JMP accesses a TSS descriptor directly in IA-32e mode, a general-protection exception occurs. 


If CALL or JMP accesses a TSS descriptor directly outside IA-32e mode, privilege levels are checked on the 
TSS descriptor. 


e. If anon-maskable interrupt (NMI), an exception, or an external interrupt accesses a task gate in the IDT in 
IA-32e mode, a general-protection exception occurs. 


f. If anon-maskable interrupt (NMI), an exception other than breakpoint exceptions (#BP) and overflow 
exceptions (#OF), or an external interrupt accesses a task gate in the IDT outside IA-32e mode, no 
privilege checks are performed. 


g. If IRET is executed with RFLAGS.NT = 1 in IA-32e mode, a general-protection exception occurs. 


h. If IRET is executed with RFLAGS.NT = 1 outside IA-32e mode, a TSS descriptor is accessed directly and no 
privilege checks are made. 


2. Checks are made on the new TSS selector (for example, that is within GDT limits). 
3. The new TSS descriptor is read. (A page fault results if a relevant GDT page is not present). 
4. The TSS descriptor is checked for proper values of type (depends on type of task switch), P bit, S bit, and limit. 


Only if checks 1-4 all pass (do not generate faults) might a VM exit occur. However, the ordering between a VM exit 
due to a task switch and a page fault resulting from accessing the old TSS or the new TSS is implementation- 
specific. Some processors may generate a page fault (instead of a VM exit due to a task switch) if accessing either 
TSS would cause a page fault. Other processors may generate a VM exit due to a task switch even if accessing 
either TSS would cause a page fault. 


If an attempt at a task switch through a task gate in the IDT causes an exception (before generating a VM exit due 
to the task switch) and that exception causes a VM exit, information about the event whose delivery that accessed 
the task gate is recorded in the IDT-vectoring information fields and information about the exception that caused 
the VM exit is recorded in the VM-exit interruption-information fields. See Section 26.2. The fact that a task gate 
was being accessed is not recorded in the VMCS. 


If an attempt at a task switch through a task gate in the IDT causes VM exit due to the task switch, information 
about the event whose delivery accessed the task gate is recorded in the IDT-vectoring fields of the VMCS. Since 
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the cause of such a VM exit is a task switch and not an interruption, the valid bit for the VM-exit interruption infor- 
mation field is 0. See Section 26.2. 


24.5 FEATURES SPECIFIC TO VMX NON-ROOT OPERATION 


Some VM-execution controls support features that are specific to VMX non-root operation. These are the VMX- 
preemption timer (Section 24.5.1) and the monitor trap flag (Section 24.5.2), translation of guest-physical 
addresses (Section 24.5.3 and Section 24.5.4), APIC virtualization (Section 24.5.5), VM functions (Section 
24.5.6), and virtualization exceptions (Section 24.5.7). 


24.5.1 VMX-Preemption Timer 


If the last VM entry was performed with the 1-setting of “activate VMX-preemption timer” VM-execution control, 
the VMX-preemption timer counts down (from the value loaded by VM entry; see Section 25.7.4) in VMX non- 
root operation. When the timer counts down to zero, it stops counting down and a VM exit occurs (see Section 
24.2). 


The VMX-preemption timer counts down at rate proportional to that of the timestamp counter (TSC). Specifically, 
the timer counts down by 1 every time bit X in the TSC changes due to a TSC increment. The value of X is in the 
range 0-31 and can be determined by consulting the VMX capability MSR IA32_VMX_MISC (see Appendix A.6). 


The VMX-preemption timer operates in the C-states CO, C1, and C2; it also operates in the shutdown and wait-for- 
SIPI states. If the timer counts down to zero in any state other than the wait-for SIPI state, the logical processor 

transitions to the CO C-state and causes a VM exit; the timer does not cause a VM exit if it counts down to zero in 

the wait-for-SIPI state. The timer is not decremented in C-states deeper than C2. 


Treatment of the timer in the case of system management interrupts (SMIs) and system-management mode 
(SMM) depends on whether the treatment of SMIs and SMM: 


e =If the default treatment of SMIs and SMM (see Section 30.14) is active, the VMX-preemption timer counts 
across an SMI to VMX non-root operation, subsequent execution in SMM, and the return from SMM via the RSM 
instruction. However, the timer can cause a VM exit only from VMX non-root operation. If the timer expires 
during SMI, in SMM, or during RSM, a timer-induced VM exit occurs immediately after RSM with its normal 
priority unless it is blocked based on activity state (Section 24.2). 


¢ If the dual-monitor treatment of SMIs and SMM (see Section 30.15) is active, transitions into and out of SMM 
are VM exits and VM entries, respectively. The treatment of the VMX-preemption timer by those transitions is 
mostly the same as for ordinary VM exits and VM entries; Section 30.15.2 and Section 30.15.4 detail some 
differences. 


24.5.2 Monitor Trap Flag 


The monitor trap flag is a debugging feature that causes VM exits to occur on certain instruction boundaries in 
VMX non-root operation. Such VM exits are called MTF VM exits. An MTF VM exit may occur on an instruction 
boundary in VMX non-root operation as follows: 


¢ If the “monitor trap flag” VM-execution control is 1 and VM entry is injecting a vectored event (see Section 
25.6.1), an MTF VM exit is pending on the instruction boundary before the first instruction following the 
VM entry. 


¢ If VM entry is injecting a pending MTF VM exit (see Section 25.6.2), an MTF VM exit is pending on the 
instruction boundary before the first instruction following the VM entry. This is the case even if the “monitor 
trap flag” VM-execution control is 0. 


¢ If the “monitor trap flag” VM-execution control is 1, VM entry is not injecting an event, and a pending event 
(e.g., debug exception or interrupt) is delivered before an instruction can execute, an MTF VM exit is pending 
on the instruction boundary following delivery of the event (or any nested exception). 


° Suppose that the “monitor trap flag” VM-execution control is 1, VM entry is not injecting an event, and the first 
instruction following VM entry is a REP-prefixed string instruction: 
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— Ifthe first iteration of the instruction causes a fault, an MTF VM exit is pending on the instruction boundary 
following delivery of the fault (or any nested exception). 


— Ifthe first iteration of the instruction does not cause a fault, an MTF VM exit is pending on the instruction 
boundary after that iteration. 


* Suppose that the “monitor trap flag” VM-execution control is 1, VM entry is not injecting an event, and the first 
instruction following VM entry is the XBEGIN instruction. In this case, an MTF VM exit is pending at the fallback 
instruction address of the XBEGIN instruction. This behavior applies regardless of whether advanced debugging 
of RTM transactional regions has been enabled (see Section 16.3.7, “RTM-Enabled Debugger Support,” of 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1). 


°® Suppose that the “monitor trap flag” VM-execution control is 1, VM entry is not injecting an event, and the first 
instruction following VM entry is neither a REP-prefixed string instruction or the XBEGIN instruction: 


— Ifthe instruction causes a fault, an MTF VM exit is pending on the instruction boundary following delivery of 
the fault (or any nested exception).! 


— Ifthe instruction does not cause a fault, an MTF VM exit is pending on the instruction boundary following 
execution of that instruction. If the instruction is INT1, INT3, or INTO, this boundary follows delivery of any 
software exception. If the instruction is INT n, this boundary follows delivery of a software interrupt. If the 
instruction is HLT, the MTF VM exit will be from the HLT activity state. 


No MTF VM exit occurs if another VM exit occurs before reaching the instruction boundary on which an MTF VM exit 
would be pending (e.g., due to an exception or triple fault). 


An MTF VM exit occurs on the instruction boundary on which it is pending unless a higher priority event takes 
precedence or the MTF VM exit is blocked due to the activity state: 


° System-management interrupts (SMIs), INIT signals, and higher priority events take priority over MTF 
VM exits. MTF VM exits take priority over debug-trap exceptions and lower priority events. 


¢ No MTF VM exit occurs if the processor is in either the shutdown activity state or wait-for-SIPI activity state. If 
a non-maskable interrupt subsequently takes the logical processor out of the shutdown activity state without 
causing a VM exit, an MTF VM exit is pending after delivery of that interrupt. 


Special treatment may apply to Intel SGX instructions or if the logical processor is in enclave mode. See Section 
38.2 for details. 


24.5.3. Translation of Guest-Physical Addresses Using EPT 


The extended page-table mechanism (EPT) is a feature that can be used to support the virtualization of physical 
memory. When EPT is in use, certain physical addresses are treated as guest-physical addresses and are not used 
to access memory directly. Instead, guest-physical addresses are translated by traversing a set of EPT paging 
structures to produce physical addresses that are used to access memory. 


Details of the EPT mechanism are given in Section 27.2. 


24.5.4 Translation of Guest-Physical Addresses Used by Intel Processor Trace 


As described in Chapter 31, Intel® Processor Trace (Intel PT) captures information about software execution using 
dedicated hardware facilities. 


Intel PT can be configured so that the trace output is written to memory using physical addresses. For example, 
when the ToPA (table of physical addresses) output mechanism is used, the IA32_RTIT_OUTPUT_BASE MSR 
contains the physical address of the base of the current ToPA. Each entry in that table contains the physical address 
of an output region in memory. When an output region becomes full, the ToPA output mechanism directs subse- 
quent trace output to the next output region as indicated in the ToPA. 


1. This item includes the cases of an invalid opcode exception—#UD— generated by the UDO, UD1, and UD2 instructions and a BOUND- 
range exceeded exception—#BR—generated by the BOUND instruction. 
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When the “Intel PT uses guest physical addresses” VM-execution control is 1, the logical processor treats the 
addresses used by Intel PT (the output addresses as well as those used to discover the output addresses) as guest- 
physical addresses, translating to physical addresses using EPT before trace output is written to memory. 


Translating these addresses through EPT implies that the trace-output mechanism may cause EPT violations and 
VM exits; details are provided in Section 24.5.4.1. Section 24.5.4.2 describes a mechanism that ensures that these 
VM exits do not cause loss of trace data. 


24.5.4.1 Guest-Physical Address Translation for Intel PT: Details 


When the “Intel PT uses guest physical addresses” VM-execution control is 1, the addresses used by Intel PT are 
treated as guest-physical addresses and translated using EPT. These addresses include the addresses of the output 
regions as well as the addresses of the ToPA entries that contain the output-region addresses. 


Translation of accesses by the trace-output process may result in EPT violations or EPT misconfigurations (Section 
27.2.3), resulting in VM exits. EPT violations resulting for the trace-output process always cause VM exits and are 
never converted to virtualization exceptions (Section 24.5.7.1). 


If no EPT violation or EPT misconfiguration occurs and if page-modification logging (Section 27.2.6) is enabled, the 
address of an output region may be added to the page-modification log. If the log is full, a page-modification log- 
full event occurs, resulting in a VM exit. 


If the “virtualize APIC accesses” VM-execution control is 1, a guest-physical address used by the trace-output 
process may be translated to an address on the APIC-access page. In this case, the access by the trace-output 
process causes an APIC-access VM exit as discussed in Section 28.4.6.1. 


24.5.4.2 Trace-Address Pre-Translation (TAPT) 


Because it buffers trace data produced by Intel PT before it is written to memory, the processor ensures that buff- 
ered data is not lost when a VM exit disables Intel PT. Specifically, the processor ensures that there is sufficient 
space left in the current output page for the buffered data. If this were not done, buffered trace data could be lost 
and the resulting trace corrupted. 


To prevent the loss of buffered trace data, the processor uses a mechanism called trace-address pre-translation 
(TAPT). With TAPT, the processor translates using EPT the guest-physical address of the current output region 
before that address would be used to write buffered trace data to memory. 


Because of TAPT, no translation (and thus no EPT violation) occurs at the time output is written to memory; the 
writes to memory use translations that were cached as part of TAPT. (The details given in Section 24.5.4.1 apply to 
TAPT.) TAPT ensures that, if a write to the output region would cause an EPT violation, the resulting VM exit is deliv- 
ered at the time of TAPT, before the region would be used. This allows software to resolve the EPT violation at that 
time and ensures that, when it is necessary to write buffered trace data to memory, that data will not be lost due 
to an EPT violation. 


TAPT (and resulting VM exits) may occur at any of the following times: 


¢ When software in VMX non-root operation enables tracing by loading the IA32_RTIT_CTL MSR to set the 
TraceEn bit, using the WRMSR instruction or the XRSTORS instruction. 


Any VM exit resulting from TAPT in this case is trap-like: the WRMSR or XRSTORS completes before the 
VM exit occurs (for example, the value of CS:RIP saved in the guest-state area of the VMCS references the 
next instruction). 


¢ At an instruction boundary when one output region becomes full and Intel PT transitions to the next output 
region. 
VM exits resulting from TAPT in this case take priority over any pending debug exceptions. Such a VM exit will 
save information about such exceptions in the guest-state area of the VMCS. 

° As part of a VM entry that enables Intel PT. See Section 25.5 for details. 


TAPT may translate not only the guest-physical address of the current output region but those of subsequent 
output regions as well. (Doing so may provide better protection of trace data.) This implies that any VM exits 
resulting from TAPT may result from the translation of output-region addresses other than that of the current 
output region. 
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24.5.5  APIC Virtualization 


APIC virtualization is a collection of features that can be used to support the virtualization of interrupts and the 
Advanced Programmable Interrupt Controller (APIC). When APIC virtualization is enabled, the processor emulates 
many accesses to the APIC, tracks the state of the virtual APIC, and delivers virtual interrupts — all in VMX non- 
root operation without a VM exit. 


Details of the APIC virtualization are given in Chapter 28. 


24.5.6 VM Functions 


AVM function is an operation provided by the processor that can be invoked from VMX non-root operation without 
a VM exit. VM functions are enabled and configured by the settings of different fields in the VMCS. Software in VMX 
non-root operation invokes a VM function with the VMFUNC instruction; the value of EAX selects the specific 

VM function being invoked. 


Section 24.5.6.1 explains how VM functions are enabled. Section 24.5.6.2 specifies the behavior of the VMFUNC 
instruction. Section 24.5.6.3 describes a specific VM function called EPTP switching. 


24.5.6.1_ Enabling VM Functions 


Software enables VM functions generally by setting the “enable VM functions” VM-execution control. A specific 
VM function is enabled by setting the corresponding VM-function control. 


Suppose, for example, that software wants to enable EPTP switching (VM function 0; see Section 23.6.14).To do 
so, it must set the “activate secondary controls” VM-execution control (bit 31 of the primary processor-based VM- 
execution controls), the “enable VM functions” VM-execution control (bit 13 of the secondary processor-based VM- 
execution controls) and the “EPTP switching” VM-function control (bit 0 of the VM-function controls). 


24.5.6.2 General Operation of the VMFUNC Instruction 


The VMFUNC instruction causes an invalid-opcode exception (#UD) if the “enable VM functions” VM-execution 
controls is 0! or the value of EAX is greater than 63 (only VM functions 0-63 can be enable). Otherwise, the instruc- 
tion causes a VM exit if the bit at position EAX is 0 in the VM-function controls (the selected VM function is not 
enabled). If such a VM exit occurs, the basic exit reason used is 59 (3BH), indicating “VMFUNC”, and the length of 
the VMFUNC instruction is saved into the VM-exit instruction-length field. If the instruction causes neither an 
invalid-opcode exception nor a VM exit due to a disabled VM function, it performs the functionality of the 

VM function specified by the value in EAX. 


Individual VM functions may perform additional fault checking (e.g., one might cause a general-protection excep- 
tion if CPL > 0). In addition, specific VM functions may include checks that might result in a VM exit. If sucha 

VM exit occurs, VM-exit information is saved as described in the previous paragraph. The specification of a 

VM function may indicate that additional VM-exit information is provided. 


The specific behavior of the EPTP-switching VM function (including checks that result in VM exits) is given in 
Section 24.5.6.3. 


24.5.6.3 EPTP Switching 


EPTP switching is VM function 0. This VM function allows software in VMX non-root operation to load a new value 
for the EPT pointer (EPTP), thereby establishing a different EPT paging-structure hierarchy (see Section 27.2 for 
details of the operation of EPT). Software is limited to selecting from a list of potential EPTP values configured in 
advance by software in VMX root operation. 


Specifically, the value of ECX is used to select an entry from the EPTP list, the 4-KByte structure referenced by the 
EPTP-list address (see Section 23.6.14; because this structure contains 512 8-Byte entries, VMFUNC causes a 
VM exit if ECX > 512). If the selected entry is a valid EPTP value (it would not cause VM entry to fail; see Section 


1. “Enable VM functions” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based V/M-execution 
controls is 0, VMX non-root operation functions as if the “enable VM functions” VM-execution control were 0. See Section 23.6.2. 
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25.2.1.1), it is stored in the EPTP field of the current VMCS and is used for subsequent accesses using guest-phys- 
ical addresses. The following pseudocode provides details: 


IF €CX > 512 
THEN VM exit; 
ELSE 
tent_EPTP := 8 bytes from EPTP-list address + 8 * ECX; 
IF tent_EPTP is not a valid EPTP value (would cause VM entry to fail if in EPTP) 
THEN VM exit; 
ELSE 
write tent_EPTP to the EPTP field in the current VMCS; 
use tent_EPTP as the new EPTP value for address translation; 
IF processor supports the 1-setting of the “EPT-violation #VE” VM-execution control 
THEN 
write ECX[15:0] to EPTP-index field in current VMCS; 
use ECX[15:0] as EPTP index for subsequent EPT-violation virtualization exceptions (see Section 24.5.7.2); 
Fl; 
Fl; 
Fl; 


Execution of the EPTP-switching VM function does not modify the state of any registers; no flags are modified. 


If the “Intel PT uses guest physical addresses” VM-execution control is 1 and IA32_RTIT_CTL.TraceEn = 1, any 
execution of the EPTP-switching VM function causes a VM exit.! 


As noted in Section 24.5.6.2, an execution of the EPTP-switching VM function that causes a VM exit (as specified 
above), uses the basic exit reason 59, indicating “VMFUNC”. The length of the VMFUNC instruction is saved into the 
VM-exit instruction-length field. No additional VM-exit information is provided. 


An execution of VMFUNC loads EPTP from the EPTP list (and thus does not cause a fault or VM exit) is called an 
EPTP-switching VMFUNC. After an EPTP-switching VMFUNC, control passes to the next instruction. The logical 
processor starts creating and using guest-physical and combined mappings associated with the new value of bits 
51:12 of EPTP; the combined mappings created and used are associated with the current VPID and PCID (these are 
not changed by VMFUNC).? If the “enable VPID” VM-execution control is 0, an EPTP-switching VMFUNC invalidates 
combined mappings associated with VPID OOOOH (for all PCIDs and for all EP4TA values, where EP4TA is the value 
of bits 51:12 of EPTP). 


Because an EPTP-switching VMFUNC may change the translation of guest-physical addresses, it may affect use of 
the guest-physical address in CR3. The EPTP-switching VMFUNC cannot itself cause a VM exit due to an EPT viola- 
tion or an EPT misconfiguration due to the translation of that guest-physical address through the new EPT paging 
structures. The following items provide details that apply if CRO.PG = 1: 


* If 32-bit paging or 4-level paging? is in use (either CR4.PAE = 0 or IA32_EFER.LMA = 1), the next memory 
access with a linear address uses the translation of the guest-physical address in CR3 through the new EPT 
paging structures. As a result, this access may cause a VM exit due to an EPT violation or an EPT misconfigu- 
ration encountered during that translation. 


¢ If PAE paging is in use (CR4.PAE = 1 and IA32_EFER.LMA = 0), an EPTP-switching VMFUNC does not load the 
four page-directory-pointer-table entries (PDPTEs) from the guest-physical address in CR3. The logical 
processor continues to use the four guest-physical addresses already present in the PDPTEs. The guest- 
physical address in CR3 is not translated through the new EPT paging structures (until some operation that 
would load the PDPTEs). 


The EPTP-switching VMFUNC cannot itself cause a VM exit due to an EPT violation or an EPT misconfiguration 
encountered during the translation of a guest-physical address in any of the PDPTEs. A subsequent memory 
access with a linear address uses the translation of the guest-physical address in the appropriate PDPTE 


1. Such a VM exit ensures the proper recording of trace data that might otherwise be lost during the change of EPT paging-structure 
hierarchy. Software handling the VM exit can change emulate the VM function and then resume the guest. 


2. If the “enable VPID” VM-execution control is 0, the current VPID is OOOOH; if CR4.PCIDE = O, the current PCID is OOOH. 
3. Earlier versions of this manual used the term “IA-32e paging” to identify 4-level paging. 
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through the new EPT paging structures. As a result, such an access may cause a VM exit due to an EPT 
violation or an EPT misconfiguration encountered during that translation. 


If an EPTP-switching VMFUNC establishes an EPTP value that enables accessed and dirty flags for EPT (by setting 
bit 6), subsequent memory accesses may fail to set those flags as specified if there has been no appropriate execu- 
tion of INVEPT since the last use of an EPTP value that does not enable accessed and dirty flags for EPT (because 
bit 6 is clear) and that is identical to the new value on bits 51:12. 


IF the processor supports the 1-setting of the “EPT-violation #VE” VM-execution control, an EPTP-switching 
VMFUNC loads the value in ECX[15:0] into to EPTP-index field in current VMCS. Subsequent EPT-violation virtual- 
ization exceptions will save this value into the virtualization-exception information area (see Section 24.5.7.2); 


24.5.7 ‘Virtualization Exceptions 
A virtualization exception is a new processor exception. It uses vector 20 and is abbreviated #VE. 


A virtualization exception can occur only in VMX non-root operation. Virtualization exceptions occur only with 
certain settings of certain VM-execution controls. Generally, these settings imply that certain conditions that would 
normally cause VM exits instead cause virtualization exceptions 


In particular, the 1-setting of the “EPT-violation #VE” VM-execution control causes some EPT violations to generate 
virtualization exceptions instead of VM exits. Section 24.5.7.1 provides the details of how the processor determines 
whether an EPT violation causes a virtualization exception or a VM exit. 


When the processor encounters a virtualization exception, it saves information about the exception to the virtual- 
ization-exception information area; see Section 24.5.7.2. 


After saving virtualization-exception information, the processor delivers a virtualization exception as it would any 
other exception; see Section 24.5.7.3 for details. 


24.5.7.1 Convertible EPT Violations 


If the “EPT-violation #VE” VM-execution control is 0 (e.g., on processors that do not support this feature), EPT 
violations always cause VM exits. If instead the control is 1, certain EPT violations may be converted to cause virtu- 
alization exceptions instead; such EPT violations are convertible. 


The values of certain EPT paging-structure entries determine which EPT violations are convertible. Specifically, 
bit 63 of certain EPT paging-structure entries may be defined to mean suppress #VE: 


* If bits 2:0 of an EPT paging-structure entry are all 0, the entry is not present.! If the processor encounters 
such an entry while translating a guest-physical address, it causes an EPT violation. The EPT violation is 
convertible if and only if bit 63 of the entry is 0. 


¢ If an EPT paging-structure entry is present, the following cases apply: 


— If the value of the EPT paging-structure entry is not supported, the entry is misconfigured. If the 
processor encounters such an entry while translating a guest-physical address, it causes an EPT misconfig- 
uration (not an EPT violation). EPT misconfigurations always cause VM exits. 


— If the value of the EPT paging-structure entry is supported, the following cases apply: 


¢ If bit 7 of the entry is 1, or if the entry is an EPT PTE, the entry maps a page. If the processor uses such 
an entry to translate a guest-physical address, and if an access to that address causes an EPT violation, 
the EPT violation is convertible if and only if bit 63 of the entry is 0. 


¢ If bit 7 of the entry is O and the entry is not an EPT PTE, the entry references another EPT paging 
structure. The processor does not use the value of bit 63 of the entry to determine whether any 
subsequent EPT violation is convertible. 


If an access to a guest-physical address causes an EPT violation, bit 63 of exactly one of the EPT paging-structure 
entries used to translate that address is used to determine whether the EPT violation is convertible: either a entry 


1. If the “mode-based execute control for EPT” VM-execution control is 1,an EPT paging-structure entry is present if any of bits 2:0 or 
bit 10 is 1. 
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that is not present (if the guest-physical address does not translate to a physical address) or an entry that maps a 
page (if it does). 


A convertible EPT violation instead causes a virtualization exception if the following all hold: 

* CRO.PE = 1; 

¢ the logical processor is not in the process of delivering an event through the IDT; 

¢ the EPT violation does not result from the output process of Intel Processor Trace (Section 24.5.4); and 
° the 32 bits at offset 4 in the virtualization-exception information area are all 0. 


Delivery of virtualization exceptions writes the value FFFFFFFFH to offset 4 in the virtualization-exception informa- 
tion area (see Section 24.5.7.2). Thus, once a virtualization exception occurs, another can occur only if software 
clears this field. 


24.5.7.2 Virtualization-Exception Information 


Virtualization exceptions save data into the virtualization-exception information area (see Section 23.6.19). 
Table 24-1 enumerates the data saved and the format of the area. 


Table 24-1. Format of the Virtualization-Exception Information Area 


Byte Offset Contents 


0 The 32-bit value that would have been saved into the VMCS as an exit reason had a VM exit occurred 
instead of the virtualization exception. For EPT violations, this value is 48 (QO000030H) 


4 FFFFFFFFH 


8 The 64-bit value that would have been saved into the VMCS as an exit qualification had a VM exit 
occurred instead of the virtualization exception 

16 The 64-bit value that would have been saved into the VMCS as a guest-linear address had a VM exit 
occurred instead of the virtualization exception 

24 The 64-bit value that would have been saved into the VMCS as a guest-physical address had a VM 
exit occurred instead of the virtualization exception 

32 The current 16-bit value of the EPTP index VM-execution control (see Section 23.6.19 and Section 
24.5.6.3) 


A VMM may allow guest software to access the virtualization-exception information area. If it does, the guest soft- 
ware may modify that memory (e.g., to clear the 32-bit value at offset 4; see Section 24.5.7.1). (This is an excep- 
tion to the general requirement given in Section 23.11.4.) 


24.5.7.3 Delivery of Virtualization Exceptions 


After saving virtualization-exception information, the processor treats a virtualization exception as it does other 
exceptions: 


¢ If bit 20 (#VE) is 1 in the exception bitmap in the VMCS, a virtualization exception causes a VM exit (see 
below). If the bit is 0, the virtualization exception is delivered using gate descriptor 20 in the IDT. 


° Virtualization exceptions produce no error code. Delivery of a virtualization exception pushes no error code on 
the stack. 


¢ With respect to double faults, virtualization exceptions have the same severity as page faults. If delivery of a 
virtualization exception encounters a nested fault that is either contributory or a page fault, a double fault 
(#DF) is generated. See Chapter 6, “Interrupt 8—Double Fault Exception (#DF)” in Intel® 64 and IA-32 Archi- 
tectures Software Developer’s Manual, Volume 3A. 


It is not possible for a virtualization exception to be encountered while delivering another exception (see 
Section 24.5.7.1). 


Vol. 3C 24-19 


VMX NON-ROOT OPERATION 


If a virtualization exception causes a VM exit directly (because bit 20 is 1 in the exception bitmap), information 
about the exception is saved normally in the VM-exit interruption information field in the VMCS (see Section 
26.2.2). Specifically, the event is reported as a hardware exception with vector 20 and no error code. Bit 12 of the 
field (NMI unblocking due to IRET) is set normally. 


If a virtualization exception causes a VM exit indirectly (because bit 20 is 0 in the exception bitmap and delivery of 
the exception generates an event that causes a VM exit), information about the exception is saved normally in the 
IDT-vectoring information field in the VMCS (see Section 26.2.4). Specifically, the event is reported as a hardware 
exception with vector 20 and no error code. 


24.6 UNRESTRICTED GUESTS 


The first processors to support VMX operation require CRO.PE and CRO.PG to be 1 in VMX operation (see Section 
22.8). This restriction implies that guest software cannot be run in unpaged protected mode or in real-address 
mode. Later processors support a VM-execution control called “unrestricted guest”.! If this control is 1, CRO.PE and 
CRO.PG may be 0 in VMX non-root operation. Such processors allow guest software to run in unpaged protected 
mode or in real-address mode. The following items describe the behavior of such software: 


¢ The MOV CRO instructions does not cause a general-protection exception simply because it would set either 
CRO.PE and CRO.PG to 0. See Section 24.3 for details. 


¢ A logical processor treats the values of CRO.PE and CRO.PG in VMX non-root operation just as it does outside 
VMX operation. Thus, if CRO.PE = 0, the processor operates as it does normally in real-address mode (for 
example, it uses the 16-bit interrupt table to deliver interrupts and exceptions). If CRO.PG = 0, the processor 
operates as it does normally when paging is disabled. 


¢ Processor operation is modified by the fact that the processor is in VMX non-root operation and by the settings 
of the VM-execution controls just as it is in protected mode or when paging is enabled. Instructions, interrupts, 
and exceptions that cause VM exits in protected mode or when paging is enabled also do so in real-address 
mode or when paging is disabled. The following examples should be noted: 


— If CRO.PG = 0, page faults do not occur and thus cannot cause VM exits. 
— If CRO.PE = 0, invalid-TSS exceptions do not occur and thus cannot cause VM exits. 


— If CRO.PE = 0, the following instructions cause invalid-opcode exceptions and do not cause VM exits: 
INVEPT, INVVPID, LLDT, LTR, SLDT, STR, VMCLEAR, VMLAUNCH, VMPTRLD, VMPTRST, VMREAD, 
VMRESUME, VMWRITE, VMXOFF, and VMXON. 


° If CRO.PG = 0, each linear address is passed directly to the EPT mechanism for translation to a physical 
address.” The guest memory type passed on to the EPT mechanism is WB (writeback). 


1. “Unrestricted guest” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VMX non-root operation functions as if the “unrestricted quest” VM-execution control were 0. See Section 23.6.2. 


2. As noted in Section 25.2.1.1, the “enable EPT” VM-execution control must be 1 if the “unrestricted quest” VM-execution control is 1. 
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Software can enter VMX non-root operation using either of the VM-entry instructions VMLAUNCH and VMRESUME. 
VMLAUNCH can be used only with a VMCS whose launch state is clear and VMRESUME can be used only with a 
VMCS whose the launch state is launched. VMLAUNCH should be used for the first VM entry after VMCLEAR; VMRE- 
SUME should be used for subsequent VM entries with the same VMCS. 


Each VM entry performs the following steps in the order indicated: 


1. Basic checks are performed to ensure that VM entry can commence 
(Section 25.1). 


2. The control and host-state areas of the VMCS are checked to ensure that they are proper for supporting VMX 
non-root operation and that the VMCS is correctly configured to support the next VM exit (Section 25.2). 


3. The following may be performed in parallel or in any order (Section 25.3): 


° The guest-state area of the VMCS is checked to ensure that, after the VM entry completes, the state of the 
logical processor is consistent with IA-32 and Intel 64 architectures. 


¢ Processor state is loaded from the guest-state area and based on controls in the VMCS. 
e Address-range monitoring is cleared. 

MSRs are loaded from the VM-entry MSR-load area (Section 25.4). 

If VMLAUNCH is being executed, the launch state of the VMCS is set to “launched.” 


If the “Intel PT uses guest physical addresses” VM-execution control is 1, trace-address pre-translation (TAPT) 
may occur (see Section 24.5.4 and Section 25.5). 


7. An event may be injected in the guest context (Section 25.6). 


Steps 1-4 above perform checks that may cause VM entry to fail. Such failures occur in one of the following three 

ways: 

¢ Some of the checks in Section 25.1 may generate ordinary faults (for example, an invalid-opcode exception). 
Such faults are delivered normally. 


¢ Some of the checks in Section 25.1 and all the checks in Section 25.2 cause control to pass to the instruction 
following the VM-entry instruction. The failure is indicated by setting RFLAGS.ZF! (if there is a current VMCS) 
or RFLAGS.CF (if there is no current VMCS). If there is a current VMCS, an error number indicating the cause of 
the failure is stored in the VM-instruction error field. See Chapter 29 for the error numbers. 


° The checks in Section 25.3 and Section 25.4 cause processor state to be loaded from the host-state area of the 
VMCS (as would be done ona VM exit). Information about the failure is stored in the VM-exit information fields. 
See Section 25.8 for details. 


EFLAGS.TF = 1 causes a VM-entry instruction to generate a single-step debug exception only if failure of one of the 
checks in Section 25.1 and Section 25.2 causes control to pass to the following instruction. A VM-entry does not 
generate a single-step debug exception in any of the following cases: (1) the instruction generates a fault; (2) 
failure of one of the checks in Section 25.3 or in loading MSRs causes processor state to be loaded from the host- 
state area of the VMCS; or (3) the instruction passes all checks in Section 25.1, Section 25.2, and Section 25.3 and 
there is no failure in loading MSRs. 


Section 30.15 describes the dual-monitor treatment of system-management interrupts (SMIs) and system- 
management mode (SMM). Under this treatment, code running in SMM returns using VM entries instead of the RSM 
instruction. A VM entry returns from SMM if it is executed in SMM and the “entry to SMM” VM-entry control is 0. 
VM entries that return from SMM differ from ordinary VM entries in ways that are detailed in Section 30.15.4. 


1. This chapter uses the notation RAX, RIP, RSP, RFLAGS, etc. for processor registers because most processors that support VMX oper- 
ation also support Intel 64 architecture. For IA-32 processors, this notation refers to the 32-bit forms of those registers (EAX, EIP, 
ESP, EFLAGS, etc.). In a few places, notation such as EAX is used to refer specifically to lower 32 bits of the indicated register. 


Vol. 3C 25-1 


VM ENTRIES 


25.1 BASIC VM-ENTRY CHECKS 


Before a VM entry commences, the current state of the logical processor is checked in the following order: 


1. If the logical processor is in virtual-8086 mode or compatibility mode, an invalid-opcode exception is 
generated. 


If the current privilege level (CPL) is not zero, a general-protection exception is generated. 
If there is no current VMCS, RFLAGS.CF is set to 1 and control passes to the next instruction. 


If there is a current VMCS but the current VMCS is a shadow VMCS (see Section 23.10), RFLAGS.CF is set to 1 
and control passes to the next instruction. 


5. If there is a current VMCS that is not a shadow VMCS, the following conditions are evaluated in order; any of 
these cause VM entry to fail: 


a. if there is MOV-SS blocking (see Table 23-3) 
b. if the VM entry is invoked by VMLAUNCH and the VMCS launch state is not clear 
c. if the VM entry is invoked by VMRESUME and the VMCS launch state is not launched 


If any of these checks fail, RFLAGS.ZF is set to 1 and control passes to the next instruction. An error number 
indicating the cause of the failure is stored in the VM-instruction error field. See Chapter 29 for the error 
numbers. 


25.2 CHECKS ON VMX CONTROLS AND HOST-STATE AREA 


If the checks in Section 25.1 do not cause VM entry to fail, the control and host-state areas of the VMCS are 
checked to ensure that they are proper for supporting VMX non-root operation, that the VMCS is correctly config- 
ured to support the next VM exit, and that, after the next VM exit, the processor’s state is consistent with the Intel 
64 and IA-32 architectures. 


VM entry fails if any of these checks fail. When such failures occur, control is passed to the next instruction, 
RFLAGS.ZF is set to 1 to indicate the failure, and the VM-instruction error field is loaded with an error number that 
indicates whether the failure was due to the controls or the host-state area (see Chapter 29). 


These checks may be performed in any order. Thus, an indication by error number of one cause (for example, host 
state) does not imply that there are not also other errors. Different processors may thus give different error 
numbers for the same VMCS. Some checks prevent establishment of settings (or combinations of settings) that are 
currently reserved. Future processors may allow such settings (or combinations) and may not perform the corre- 
sponding checks. The correctness of software should not rely on VM-entry failures resulting from the checks docu- 
mented in this section. 


The checks on the controls and the host-state area are presented in Section 25.2.1 through Section 25.2.4. These 
sections reference VMCS fields that correspond to processor state. Unless otherwise stated, these references are to 
fields in the host-state area. 


25.2.1 Checks on VMX Controls 


This section identifies VM-entry checks on the VMX control fields. 


25.2.1.1 VM-Execution Control Fields 


VM entries perform the following checks on the VM-execution control fields:! 


¢ Reserved bits in the pin-based VM-execution controls must be set properly. Software may consult the VMX 
capability MSRs to determine the proper settings (see Appendix A.3.1). 


1. If the “activate secondary controls” primary processor-based VM-execution control is 0, VM entry operates as if each secondary pro- 
cessor-based VM-execution control were O. Similarly, if the “activate tertiary controls” primary processor-based VM-execution con- 
trol is 0, VM entry operates as if each tertiary processor-based VM-execution control were 0. 
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Reserved bits in the primary processor-based VM-execution controls must be set properly. Software may 
consult the VMX capability MSRs to determine the proper settings (see Appendix A.3.2). 


If the “activate secondary controls” primary processor-based VM-execution control is 1, reserved bits in the 
secondary processor-based VM-execution controls must be cleared. Software may consult the VMX capability 
MSRs to determine which bits are reserved (see Appendix A.3.3). 


If the “activate secondary controls” primary processor-based VM-execution control is 0 (or if the processor 
does not support the 1-setting of that control), no checks are performed on the secondary processor-based 
VM-execution controls. The logical processor operates as if all the secondary processor-based VM-execution 
controls were 0. 


If the “activate tertiary controls” primary processor-based VM-execution control is 1, reserved bits in the 
tertiary processor-based VM-execution controls must be cleared. Software may consult the VMX capability 
MSRs to determine which bits are reserved (see Appendix A.3.4). 


If the “activate tertiary controls” primary processor-based VM-execution control is 0 (or if the processor does 
not support the 1-setting of that control), no checks are performed on the tertiary processor-based VM- 
execution controls. The logical processor operates as if all the tertiary processor-based VM-execution controls 
were 0. 


The CR3-target count must not be greater than 4. Future processors may support a different number of CR3- 
target values. Software should read the VMX capability MSR IA32_VMX_MISC to determine the number of 
values supported (see Appendix A.6). 


If the “use I/O bitmaps” VM-execution control is 1, bits 11:0 of each I/O-bitmap address must be 0. Neither 
address should set any bits beyond the processor’s physical-address width.!/2 


If the “use MSR bitmaps” VM-execution control is 1, bits 11:0 of the MSR-bitmap address must be 0. The 
address should not set any bits beyond the processor’s physical-address width. 


If the “use TPR shadow” VM-execution control is 1, the virtual-APIC address must satisfy the following checks: 
— Bits 11:0 of the address must be 0. 
— The address should not set any bits beyond the processor’s physical-address width.* 


If all of the above checks are satisfied and the “use TPR shadow” VM-execution control is 1, bytes 3:1 of VTPR 
(see Section 28.1.1) may be cleared (behavior may be implementation-specific). 


The clearing of these bytes may occur even if the VM entry fails. This is true either if the failure causes control 
to pass to the instruction following the VM-entry instruction or if it causes processor state to be loaded from 
the host-state area of the VMCS. 


If the “use TPR shadow” VM-execution control is 1 and the “virtual-interrupt delivery” VM-execution control is 
0, bits 31:4 of the TPR threshold VM-execution control field must be 0.5 


The following check is performed if the “use TPR shadow” VM-execution control is 1 and the “virtualize APIC 
accesses” and “virtual-interrupt delivery” VM-execution controls are both 0: the value of bits 3:0 of the TPR 
threshold VM-execution control field should not be greater than the value of bits 7:4 of VTPR (see Section 
28.1.1). 


If the “NMI exiting” VM-execution control is 0, the “virtual NMIs” VM-execution control must be 0. 
If the “virtual NMIs” VM-execution control is 0, the “NMI-window exiting” VM-execution control must be 0. 


If the “virtualize APIC-accesses” VM-execution control is 1, the APIC-access address must satisfy the following 
checks: 


— Bits 11:0 of the address must be 0. 


ne wn 


. Software can determine a processor's physical-address width by executing CPUID with BOOOOOOSH in EAX. The physical-address 


width is returned in bits 7:0 of EAX. 

If 1A32_VMX_BASIC[48] is read as 1, these addresses must not set any bits in the range 63:32; see Appendix A.1. 
If 1A32_VMX_BASIC[48] is read as 1, this address must not set any bits in the range 63:32; see Appendix A.1. 

If 1A32_VMX_BASIC[48] is read as 1, this address must not set any bits in the range 63:32; see Appendix A.1. 


‘Nirtual-interrupt delivery” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-exe- 
cution controls is 0, VM entry functions as if the “virtual-interrupt delivery” VM-execution control were 0. See Section 23.6.2. 
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— The address should not set any bits beyond the processor’s physical-address width.? 


e If the “use TPR shadow” VM-execution control is 0, the following VM-execution controls must also be 0: 


“virtualize x2APIC mode”, “APIC-register virtualization”, and “virtual-interrupt delivery”.2 


e If the “virtualize x2APIC mode” VM-execution control is 1, the “virtualize APIC accesses” VM-execution control 
must be 0. 


e If the “virtual-interrupt delivery” VM-execution control is 1, the “external-interrupt exiting” VM-execution 
control must be 1. 

* If the “process posted interrupts” VM-execution control is 1, the following must be true:? 

— The “virtual-interrupt delivery” VM-execution control is 1. 

— The “acknowledge interrupt on exit” VM-exit control is 1. 

— The posted-interrupt notification vector has a value in the range 0-255 (bits 15:8 are all 0). 

— Bits 5:0 of the posted-interrupt descriptor address are all 0. 


— The posted=interuat descriptor address does not set any bits beyond the processor's physical-address 
width. 


e Ifthe “enable VPID” VM-execution control is 1, the value of the VPID VM-execution control field must not be 
OOOOH. 


e If the “enable EPT” VM-execution control is 1, the EPTP VM-execution control field (see Table 23-9 in Section 
23.6.11) must satisfy the following checks:® 


— The EPT memory type (bits 2:0) must be a value supported by the processor as indicated in the 
IA32_VMX_EPT_VPID_CAP MSR (see Appendix A.10). 


— Bits 5:3 (1 less than the EPT page-walk length) must be 3, indicating an EPT page-walk length of 4; see 
Section 27.2.2. 


— Bit 6 (enable bit for accessed and dirty flags for EPT) must be 0 if bit 21 of the IA32_VMX_EPT_VPID_CAP 
MSR (see Appendix A.10) is read as 0, indicating that the processor does not support accessed and dirty 
flags for EPT. 


— Reserved bits 11:7 and 63:N (where N is the processor’s physical-address width) must all be 0. 


e If the “enable PML” VM-execution control is 1, the “enable EPT” VM-execution control must also be 1.7 In 
addition, the PML address must satisfy the following checks: 


— Bits 11:0 of the address must be 0. 
— The address should not set any bits beyond the processor's physical-address width. 


e If either the “unrestricted guest” VM-execution control or the “mode-based execute control for EPT” VM- 
execution control is 1, the “enable EPT” VM-execution control must also be 1.8 


If [A32_VMX_BASIC[48] is read as 1, this address must not set any bits in the range 63:32; see Appendix A.1. 


2. “Virtualize x2APIC mode” and “APIC-register virtualization” are secondary processor-based V/M-execution controls. If bit 31 of the 
primary processor-based VM-execution controls is 0, VM entry functions as if these controls were O. See Section 23.6.2. 


3. “Process posted interrupts” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based V/M-exe- 
cution controls is 0, VM entry functions as if the “process posted interrupts” VM-execution control were O. See Section 23.6.2. 


4. If |A32_VMX_BASIC[48] is read as 1, this address must not set any bits in the range 63:32; see Appendix A.1. 


5. “Enable VPID” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution controls 
is 0, VM entry functions as if the “enable VPID” VM-execution control were O. See Section 23.6.2. 


6. “Enable EPT” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution controls 
is 0, VM entry functions as if the “enable EPT” VM-execution control were O. See Section 23.6.2. 


7. “Enable PML” and “enable EPT” are both secondary processor-based VM-execution controls. If bit 31 of the primary processor-based 
V/M-execution controls is 0, VM entry functions as if both these controls were O. See Section 23.6.2. 


8. All these controls are secondary processor-based VM-execution controls. If bit 31 of the primary processor-based VM-execution con- 
trols is 0, VM entry functions as if all these controls were 0. See Section 23.6.2. 
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If the “sub-page write permissions for EPT” VM-execution control is 1, the “enable EPT” VM-execution control 
must also be 1.1 In addition, the SPPTP VM-execution control field (see Table 23-11 in Section 23.6.21) must 
satisfy the following checks: 


— Bits 11:0 of the address must be 0. 
— The address should not set any bits beyond the processor’s physical-address width. 


If the “enable VM functions” processor-based VM-execution control is 1, reserved bits in the VM-function 
controls must be clear.* Software may consult the VMX capability MSRs to determine which bits are reserved 
(see Appendix A.11). In addition, the following check is performed based on the setting of bits in the VM- 
function controls (see Section 23.6.14): 


— If “EPTP switching” VM-function control is 1, the “enable EPT” VM-execution control must also be 1. In 
addition, the EPTP-list address must satisfy the following checks: 


¢ Bits 11:0 of the address must be 0. 
¢ The address must not set any bits beyond the processor's physical-address width. 


If the “enable VM functions” processor-based VM-execution control is 0, no checks are performed on the VM- 
function controls. 


If the “VMCS shadowing” VM-execution control is 1, the VMREAD-bitmap and VMWRITE-bitmap addresses 
must each satisfy the following checks:? 


— Bits 11:0 of the address must be 0. 
— The address must not set any bits beyond the processor’s physical-address width. 


If the “EPT-violation #VE” VM-execution control is 1, the virtualization-exception information address must 
satisfy the following checks:4 


— Bits 11:0 of the address must be 0. 
— The address must not set any bits beyond the processor’s physical-address width. 


If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of 
VM entry, the “load IA32_RTIT_CTL” VM-entry control must be 0. 


If the “Intel PT uses guest physical addresses” VM-execution control is 1, the following controls must also be 1: 
the “enable EPT” VM-execution control; the “load IA32_RTIT_CTL” VM-entry control; and the “clear 
IA32_RTIT_CTL” VM-exit control.° 


If the “use TSC scaling” VM-execution control is 1, the TSC-multiplier must not be zero.® 


25.2.1.2 VM-Exit Control Fields 


VM entries perform the following checks on the VM-exit control fields. 


Reserved bits in the VM-exit controls must be set properly. Software may consult the VMX capability MSRs to 
determine the proper settings (see Appendix A.4). 


— 


. “Sub-page write permissions for EPT” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based 


\V/M-execution controls is 0, VM entry functions as if the “sub-page write permissions for EPT” VM-execution control were 0. See 
Section 23.6.2. 


“Enable VM functions” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “enable VM functions” VM-execution control were O. See Section 23.6.2. 

“WMCS shadowing” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “V/MCS shadowing” VM-execution control were 0. See Section 23.6.2. 

“EPT-violation #VE" is a secondary processor-based V/M-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “EPT-violation #VE” VM-execution control were 0. See Section 23.6.2. 

“Intel PT uses guest physical addresses” is a secondary processor-based VM-execution control. If bit 31 of the primary processor- 
based VM-execution controls is 0, VM entry functions as if the “Intel PT uses quest physical addresses” VM-execution control were 
0. See Section 23.6.2. 

“Use TSC scaling” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “use TSC scaling” VM-execution control were O. See Section 23.6.2. 
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° If the “activate VMX-preemption timer” VM-execution control is 0, the “save VMX-preemption timer value” VM- 
exit control must also be 0. 


¢ The following checks are performed for the VM-exit MSR-store address if the VM-exit MSR-store count field is 
non-zero: 


— The lower 4 bits of the VM-exit MSR-store address must be 0. The address should not set any bits beyond 
the processor’s physical-address width. 


— The address of the last byte in the VM-exit MSR-store area should not set any bits beyond the processor’s 
physical-address width. The address of this last byte is VM-exit MSR-store address + (MSR count * 16) - 1. 
(The arithmetic used for the computation uses more bits than the processor’s physical-address width.) 


If IA32_VMX_BASIC[48] is read as 1, neither address should set any bits in the range 63:32; see Appendix 
A.1. 


¢ The following checks are performed for the VM-exit MSR-load address if the VM-exit MSR-load count field is 
non-zero: 


— The lower 4 bits of the VM-exit MSR-load address must be 0. The address should not set any bits beyond 
the processor’s physical-address width. 


— The address of the last byte in the VM-exit MSR-load area should not set any bits beyond the processor's 
physical-address width. The address of this last byte is VM-exit MSR-load address + (MSR count * 16) - 1. 
(The arithmetic used for the computation uses more bits than the processor’s physical-address width.) 


If IA32_VMX_BASIC[48] is read as 1, neither address should set any bits in the range 63:32; see Appendix 
A.1. 


25.2.1.3. \VM-Entry Control Fields 
VM entries perform the following checks on the VM-entry control fields. 


¢ Reserved bits in the VM-entry controls must be set properly. Software may consult the VMX capability MSRs to 
determine the proper settings (see Appendix A.5). 


¢ Fields relevant to VM-entry event injection must be set properly. These fields are the VM-entry interruption- 
information field (see Table 23-15 in Section 23.8.3), the VM-entry exception error code, and the VM-entry 
instruction length. If the valid bit (bit 31) in the VM-entry interruption-information field is 1, the following must 
hold: 


— The field’s interruption type (bits 10:8) is not set to a reserved value. Value 1 is reserved on all logical 
processors; value 7 (other event) is reserved on logical processors that do not support the 1-setting of the 
“monitor trap flag” VM-execution control. 


— The field’s vector (bits 7:0) is consistent with the interruption type: 
¢ If the interruption type is non-maskable interrupt (NMI), the vector is 2. 
¢ If the interruption type is hardware exception, the vector is at most 31. 
¢ If the interruption type is other event, the vector is 0 (pending MTF VM exit). 


— The field's deliver-error-code bit (bit 11) is 1 if each of the following holds: (1) the interruption type is 
hardware exception; (2) bit 0 (corresponding to CRO.PE) is set in the CRO field in the guest-state area; 
(3) IA32_VMX_BASIC[56] is read as 0 (see Appendix A.1); and (4) the vector indicates one of the following 
exceptions: #DF (vector 8), #TS (10), #NP (11), #SS (12), #GP (13), #PF (14), or #AC (17). 


— The field's deliver-error-code bit is 0 if any of the following holds: (1) the interruption type is not hardware 
exception; (2) bit 0 is clear in the CRO field in the guest-state area; or (3) IA32_VMX_BASIC[56] is read as 
0 and the vector is in one of the following ranges: 0-7, 9, 15, 16, or 18-31. 


— Reserved bits in the field (30:12) are 0. 
— Ifthe deliver-error-code bit (bit 11) is 1, bits 31:16 of the VM-entry exception error-code field are 0. 


1. Software can determine a processor's physical-address width by executing CPUID with BOO00008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 
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— Ifthe interruption type is software interrupt, software exception, or privileged software exception, the 
VM-entry instruction-length field is in the range 0-15. A VM-entry instruction length of 0 is allowed only if 
IA32_VMX_MISC[30] is read as 1; see Appendix A.6. 


The following checks are performed for the VM-entry MSR-load address if the VM-entry MSR-load count field is 
non-zero: 


— The lower 4 bits of the VM-entry MSR-load address must be 0. The address should not set any bits beyond 
the processor's physical-address width. 


— The address of the last byte in the VM-entry MSR-load area should not set any bits beyond the processor’s 
physical-address width. The address of this last byte is VM-entry MSR-load address + (MSR count * 16) - 
1. (The arithmetic used for the computation uses more bits than the processor’s physical-address width.) 


If IA32_VMX_BASIC[48] is read as 1, neither address should set any bits in the range 63:32; see Appendix 
A.1. 


If the processor is not in SMM, the “entry to SMM” and “deactivate dual-monitor treatment” VM-entry controls 
must be 0. 


The “entry to SMM” and “deactivate dual-monitor treatment” VM-entry controls cannot both be 1. 


25.2.2 Checks on Host Control Registers, MSRs, and SSP 


The following checks are performed on fields in the host-state area that correspond to control registers and MSRs: 


The CRO field must not set any bit to a value not supported in VMX operation (see Section 22.8).2 
The CR4 field must not set any bit to a value not supported in VMX operation (see Section 22.8). 
If bit 23 in the CR4 field (corresponding to CET) is 1, bit 16 in the CRO field (WP) must also be 1. 


On processors that support Intel 64 architecture, the CR3 field must be such that bits 63:52 and bits in the 
range 51:32 beyond the processor’s physical-address width must be 0.34 

On processors that support Intel 64 architecture, the IA32_SYSENTER_ESP field and the IA32_SYSENTER_EIP 
field must each contain a canonical address. 

If the “load IA32_PERF_GLOBAL_CTRL” VM-exit control is 1, bits reserved in the IA32_PERF_GLOBAL_CTRL 
MSR must be 0 in the field for that register (see Figure 18-3). 

If the “load IA32_PAT” VM-exit control is 1, the value of the field for the IA32_PAT MSR must be one that could 
be written by WRMSR without fault at CPL 0. Specifically, each of the 8 bytes in the field must have one of the 
values 0 (UC), 1 (WC), 4 (WT), 5 (WP), 6 (WB), or 7 (UC-). 

If the “load IA32_EFER” VM-exit control is 1, bits reserved in the IA32_EFER MSR must be 0 in the field for that 


register. In addition, the values of the LMA and LME bits in the field must each be that of the “host address- 
space size” VM-exit control. 


If the “load CET state” VM-exit control is 1, the IA32_S_ CET field must not set any bits reserved in the 
IA32_S_CET MSR, and bit 10 (corresponding to SUPPRESS) and bit 11 (TRACKER) in the field cannot both be 
set. 


If the “load CET state” VM-exit control is 1, bits 1:0 must be 0 in the SSP field. 
If the “load PKRS” VM-exit control is 1, bits 63:32 must be 0 in the IA32_PKRS field. 


Software can determine a processor's physical-address width by executing CPUID with BOOOOOO8H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


The bits corresponding to CRO.NW (bit 29) and CRO.CD (bit 30) are never checked because the values of these bits are not changed 
by VM exit; see Section 26.5.1. 


Software can determine a processor's physical-address width by executing CPUID with 80O000008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


Bit 63 of the CR3 field in the host-state area must be O. This is true even though, If CR4.PCIDE = 1, bit 63 of the source operand to 
MOV to CR3 is used to determine whether cached translation information is invalidated. 
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25.2.3. Checks on Host Segment and Descriptor-Table Registers 


The following checks are performed on fields in the host-state area that correspond to segment and descriptor- 
table registers: 


¢ In the selector field for each of CS, SS, DS, ES, FS, GS and TR, the RPL (bits 1:0) and the TI flag (bit 2) must 
be 0. 


¢ The selector fields for CS and TR cannot be OOOOH. 
¢ The selector field for SS cannot be OOOOH if the “host address-space size” VM-exit control is 0. 


* On processors that support Intel 64 architecture, the base-address fields for FS, GS, GDTR, IDTR, and TR must 
contain canonical addresses. 


25.2.4 Checks Related to Address-Space Size 


On processors that support Intel 64 architecture, the following checks related to address-space size are performed 
on VMX controls and fields in the host-state area: 


¢ If the logical processor is outside IA-32e mode (if IA32_EFER.LMA = 0) at the time of VM entry, the following 
must hold: 


— The “IA-32e mode guest” VM-entry control is 0. 
— The “host address-space size” VM-exit control is 0. 


¢ If the logical processor is in IA-32e mode (if IA32_EFER.LMA = 1) at the time of VM entry, the “host address- 
space size” VM-exit control must be 1. 


e If the “host address-space size” VM-exit control is 0, the following must hold: 

— The “IA-32e mode guest” VM-entry control is 0. 

— Bit 17 of the CR4 field (corresponding to CR4.PCIDE) is 0. 

— Bits 63:32 in the RIP field are 0. 

— Ifthe “load CET state” VM-exit control is 1, bits 63:32 in the IA32_S_CET field and in the SSP field are 0. 
e If the “host address-space size” VM-exit control is 1, the following must hold: 

— Bit 5 of the CR4 field (corresponding to CR4.PAE) is 1. 

— The RIP field contains a canonical address. 


— Ifthe “load CET state” VM-exit control is 1, the IA32_S_CET field and the SSP field contain canonical 
addresses. 


e If the “load CET state” VM-exit control is 1, the IA32_INTERRUPT_SSP_TABLE_ADDR field contains a canonical 
address. 


On processors that do not support Intel 64 architecture, checks are performed to ensure that the “IA-32e mode 
guest” VM-entry control and the “host address-space size” VM-exit control are both 0. 


25.3 CHECKING AND LOADING GUEST STATE 


If all checks on the VMX controls and the host-state area pass (see Section 25.2), the following operations take 
place concurrently: (1) the guest-state area of the VMCS is checked to ensure that, after the VM entry completes, 
the state of the logical processor is consistent with IA-32 and Intel 64 architectures; (2) processor state is loaded 
from the guest-state area or as specified by the VM-entry control fields; and (3) address-range monitoring is 
cleared. 


Because the checking and the loading occur concurrently, a failure may be discovered only after some state has 
been loaded. For this reason, the logical processor responds to such failures by loading state from the host-state 
area, as it would for a VM exit. See Section 25.8. 
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25.3.1 Checks on the Guest State Area 


This section describes checks performed on fields in the guest-state area. These checks may be performed in any 
order. Some checks prevent establishment of settings (or combinations of settings) that are currently reserved. 

Future processors may allow such settings (or combinations) and may not perform the corresponding checks. The 
correctness of software should not rely on VM-entry failures resulting from the checks documented in this section. 


The following subsections reference fields that correspond to processor state. Unless otherwise stated, these refer- 
ences are to fields in the guest-state area. 


25.3.1.1 Checks on Guest Control Registers, Debug Registers, and MSRs 
The following checks are performed on fields in the guest-state area corresponding to control registers, debug 
registers, and MSRs: 


¢ The CRO field must not set any bit to a value not supported in VMX operation (see Section 22.8). The following 
are exceptions: 


— Bito Cores pending to CRO.PE) and bit 31 (PG) are not checked if the “unrestricted guest” VM-execution 
control is 1. 


— Bit 29 (corresponding to CRO.NW) and bit 30 (CD) are never checked because the values of these bits are 
not changed by VM entry; see Section 25.3.2.1. 


* If bit 31 in the CRO field (corresponding to PG) is 1, bit 0 in that field (PE) must also be 1.2 
° The CR4 field must not set any bit to a value not supported in VMX operation (see Section 22.8). 
¢ If bit 23 in the CR4 field (corresponding to CET) is 1, bit 16 in the CRO field (WP) must also be 1. 


¢ If the “load debug controls” VM-entry control is 1, bits reserved in the IA32_DEBUGCTL MSR must be 0 in the 
field for that register. The first processors to support the virtual-machine extensions supported only the 1- 
setting of this control and thus performed this check unconditionally. 


¢ The following checks are performed on processors that support Intel 64 architecture: 


— If the “IA-32e mode guest” VM-entry control is 1, bit 31 in the CRO field (corresponding to CRO.PG) and 
bit 5 in the CR4 field (corresponding to CR4.PAE) must each be 1.° 

— Ifthe “IA-32e mode guest” VM-entry control is 0, bit 17 in the CR4 field (corresponding to CR4.PCIDE) 
must be 0. 


— The CR3 field must be such that bits 63:52 and bits in the range 51:32 beyond the processor's physical- 
address width are 0.4" 


— Ifthe “load debug controls” VM-entry control is 1, bits 63:32 in the DR7 field must be 0. The first 
processors to support the virtual-machine extensions supported only the 1-setting of this control and thus 
performed this check unconditionally (if they supported Intel 64 architecture). 


— The IA32_SYSENTER_ESP field and the IA32_SYSENTER_EIP field must each contain a canonical address. 


— Ifthe “load CET state” VM-entry control is 1, the IA32_S_CET field and the 
IA32_INTERRUPT_SSP_TABLE_ADDR field must contain canonical addresses. 


e If the “load IA32_PERF_GLOBAL_CTRL” VM-entry control is 1, bits reserved in the IA32_PERF_GLOBAL_CTRL 
MSR must be 0 in the field for that register (see Figure 18-3). 


1. “Unrestricted quest” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “unrestricted quest” VM-execution control were 0. See Section 23.6.2. 


2. If the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PE must be 1 in VMX operation, bit 0 in the CRO field must be 1 
unless the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are both 1. 


3. If the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, bit 31 in the CRO field must be 1 
unless the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are both 1. 


4, Software can determine a processor's physical-address width by executing CPUID with 8OO00008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


5. Bit 63 of the CR3 field in the quest-state area must be O. This is true even though, If CR4.PCIDE = 1, bit 63 of the source operand to 
MOV to CR3 is used to determine whether cached translation information is invalidated. 
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If the “load IA32_ PAT” VM-entry control is 1, the value of the field for the IA32_PAT MSR must be one that could 
be written by WRMSR without fault at CPL 0. Specifically, each of the 8 bytes in the field must have one of the 
values 0 (UC), 1 (WC), 4 (WT), 5 (WP), 6 (WB), or 7 (UC-). 

If the “load IA32_EFER” VM-entry control is 1, the following checks are performed on the field for the 
IA32_EFER MSR: 

— Bits reserved in the IA32_EFER MSR must be 0. 


— Bit 10 (corresponding to IA32_EFER.LMA) must equal the value of the “IA-32e mode guest” VM-entry 
control. It must also be identical to bit 8 (LME) if bit 31 in the CRO field (corresponding to CRO.PG) is 1.1 


If the “load IA32_BNDCFGS” VM-entry control is 1, the following checks are performed on the field for the 
IA32_BNDCFGS MSR: 


— Bits reserved in the IA32_BNDCFGS MSR must be 0. 
— The linear address in bits 63:12 must be canonical. 


If the “load IA32_RTIT_CTL” VM-entry control is 1, bits reserved in the IA32_RTIT_CTL MSR must be 0 in the 
field for that register (see Table 31-6). 


If the “load CET state” VM-entry control is 1, the IA32_S_CET field must not set any bits reserved in the 
IA32_S_CET MSR, and bit 10 (corresponding to SUPPRESS) and bit 11 (TRACKER) of the field cannot both be 
set. 


If the “load PKRS” VM-entry control is 1, bits 63:32 must be 0 in the IA32_PKRS field. 


25.3.1.2 Checks on Guest Segment Registers 


This section specifies the checks on the fields for CS, SS, DS, ES, FS, GS, TR, and LDTR. The following terms are 
used in defining these checks: 


The guest will be virtual-8086 if the VM flag (bit 17) is 1 in the RFLAGS field in the guest-state area. 


The guest will be IA-32e mode if the “IA-32e mode guest” VM-entry control is 1. (This is possible only on 
processors that support Intel 64 architecture.) 

Any one of these registers is said to be usable if the unusable bit (bit 16) is 0 in the access-rights field for that 
register. 


The following are the checks on these fields: 


Selector fields. 
— TR. The TI flag (bit 2) must be 0. 
— LDTR. If LDTR is usable, the TI flag (bit 2) must be 0. 


— SS. If the guest will not be virtual-8086 and the “unrestricted guest” VM-execution control is 0, the RPL 
(bits 1:0) must equal the RPL of the selector field for CS.? 


Base-address fields. 


— CS, SS, DS, ES, FS, GS. If the guest will be virtual-8086, the address must be the selector field shifted left 
4 bits (multiplied by 16). 


— The following checks are performed on processors that support Intel 64 architecture: 
¢ TR, FS, GS. The address must be canonical. 
¢ LDTR. If LDTR is usable, the address must be canonical. 
¢ CS. Bits 63:32 of the address must be zero. 
¢ SS, DS, ES. If the register is usable, bits 63:32 of the address must be zero. 


If the capability MSR 1IA32_VMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, bit 31 in the CRO field must be 1 
unless the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are both 1. 


“Unrestricted quest” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “unrestricted guest” VM-execution control were O. See Section 23.6.2. 
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¢ Limit fields for CS, SS, DS, ES, FS, GS. If the guest will be virtual-8086, the field must be OOOOFFFFH. 
° Access-rights fields. 
— CS, SS, DS, ES, FS, GS. 
¢ If the guest will be virtual-8086, the field must be OOOOOOF3H. This implies the following: 


Bits 3:0 (Type) must be 3, indicating an expand-up read/write accessed data segment. 
Bit 4 (S) must be 1. 

Bits 6:5 (DPL) must be 3. 

Bit 7 (P) must be 1. 


Bits 11:8 (reserved), bit 12 (software available), bit 13 (reserved/L), bit 14 (D/B), bit 15 (G), 
bit 16 (unusable), and bits 31:17 (reserved) must all be 0. 


¢ If the guest will not be virtual-8086, the different sub-fields are considered separately: 


Bits 3:0 (Type). 


¢ CS. The values allowed depend on the setting of the “unrestricted guest” VM-execution 
control: 


— If the control is 0, the Type must be 9, 11, 13, or 15 (accessed code segment). 


— If the control is 1, the Type must be either 3 (read/write accessed expand-up data 
segment) or one of 9, 11, 13, and 15 (accessed code segment). 


¢ SS. If SS is usable, the Type must be 3 or 7 (read/write, accessed data segment). 
¢ DS, ES, FS, GS. The following checks apply if the register is usable: 
— Bit 0 of the Type must be 1 (accessed). 
— If bit 3 of the Type is 1 (code segment), then bit 1 of the Type must be 1 (readable). 
Bit 4 (S). If the register is CS or if the register is usable, S must be 1. 
Bits 6:5 (DPL). 
* CS. 


— If the Type is 3 (read/write accessed expand-up data segment), the DPL must be 0. The 
Type can be 3 only if the “unrestricted guest” VM-execution control is 1. 


— If the Type is 9 or 11 (non-conforming code segment), the DPL must equal the DPL in the 
access-rights field for SS. 


— If the Type is 13 or 15 (conforming code segment), the DPL cannot be greater than the 
DPL in the access-rights field for SS. 


— If the “unrestricted guest” VM-execution control is 0, the DPL must equal the RPL from the 
selector field. 


— The DPL must be 0 either if the Type in the access-rights field for CS is 3 (read/write 
aebesere expand-up data segment) or if bit 0 in the CRO field (corresponding to CRO.PE) is 
0. 


¢ DS, ES, FS, GS. The DPL cannot be less than the RPL in the selector field if (1) the 
“unrestricted guest” VM-execution control is 0; (2) the register is usable; and (3) the Type in 
the access-rights field is in the range 0 - 11 (data segment or non-conforming code segment). 


Bit 7 (P). If the register is CS or if the register is usable, P must be 1. 
Bits 11:8 (reserved). If the register is CS or if the register is usable, these bits must all be 0. 


1. The following apply if either the “unrestricted quest” VM-execution control or bit 31 of the primary processor-based VM-execution 
controls is 0: (1) bit O in the CRO field must be 1 if the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PE must be 1 in 
VMX operation; and (2) the Type in the access-rights field for CS cannot be 3. 
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— Bit 14 (D/B). For CS, D/B must be 0 if the guest will be IA-32e mode and the L bit (bit 13) in the 
access-rights field is 1. 


— Bit 15 (G). The following checks apply if the register is CS or if the register is usable: 
¢ If any bit in the limit field in the range 11:0 is 0, G must be 0. 
¢ If any bit in the limit field in the range 31:20 is 1, G must be 1. 
— Bits 31:17 (reserved). If the register is CS or if the register is usable, these bits must all be 0. 
— TR. The different sub-fields are considered separately: 
¢ Bits 3:0 (Type). 


— If the guest will not be IA-32e mode, the Type must be 3 (16-bit busy TSS) or 11 (32-bit busy 
TSS). 


— If the guest will be IA-32e mode, the Type must be 11 (64-bit busy TSS). 
¢ Bit 4(S). S must be 0. 
¢ Bit 7 (P). P must be 1. 
¢ Bits 11:8 (reserved). These bits must all be 0. 
* Bit 15 (G). 
— If any bit in the limit field in the range 11:0 is 0, G must be 0. 
— If any bit in the limit field in the range 31:20 is 1, G must be 1. 
¢ Bit 16 (Unusable). The unusable bit must be 0. 
¢ Bits 31:17 (reserved). These bits must all be 0. 
— LDTR. The following checks on the different sub-fields apply only if LDTR is usable: 
¢ Bits 3:0 (Type). The Type must be 2 (LDT). 
¢ Bit 4(S). S must be 0. 
¢ Bit 7 (P). P must be 1. 
¢ Bits 11:8 (reserved). These bits must all be 0. 
* Bit 15 (G). 
— If any bit in the limit field in the range 11:0 is 0, G must be 0. 
— If any bit in the limit field in the range 31:20 is 1, G must be 1. 
¢ Bits 31:17 (reserved). These bits must all be 0. 


25.3.1.3 Checks on Guest Descriptor-Table Registers 

The following checks are performed on the fields for GDTR and IDTR: 

* On processors that support Intel 64 architecture, the base-address fields must contain canonical addresses. 
¢ Bits 31:16 of each limit field must be 0. 


25.3.1.4 Checks on Guest RIP, RFLAGS, and SSP 


The following checks are performed on fields in the guest-state area corresponding to RIP, RFLAGS, and SSP 
(shadow-stack pointer): 


° RIP. The following checks are performed on processors that support Intel 64 architecture: 


— Bits 63:32 must be 0 if the “IA-32e mode guest” VM-entry control is 0 or if the L bit (bit 13) in the access- 
rights field for CS is 0. 


— Ifthe processor supports N < 64 linear-address bits, bits 63:N must be identical if the “IA-32e mode guest” 
VM-entry control is 1 and the L bit in the access-rights field for CS is 1.1 (No check applies if the processor 
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supports 64 linear-address bits.) The guest RIP value is not required to be canonical; the value of bit N-1 
may differ from that of bit N. 


° RFLAGS. 


— Reserved bits 63:22 (bits 31:22 on processors that do not support Intel 64 architecture), bit 15, bit 5 and 
bit 3 must be 0 in the field, and reserved bit 1 must be 1. 


— The VM flag (bit 17) must be 0 either if the “IA-32e mode guest” VM-entry control is 1 or if bit 0 in the CRO 
field (corresponding to CRO.PE) is 0.1 


— The IF flag (RFLAGS[bit 9]) must be 1 if the valid bit (bit 31) in the VM-entry interruption-information field 
is 1 and the interruption type (bits 10:8) is external interrupt. 


° SSP. The following checks are performed if the “load CET state” VM-entry control is 1 
— Bits 1:0 must be 0. 


— Ifthe processor supports the Intel 64 architecture, bits 63:N must be identical, where N is the CPU’s 
maximum linear-address width. (This check does not apply if the processor supports 64 linear-address 
bits.) The guest SSP value is not required to be canonical; the value of bit N-1 may differ from that of bit N. 


25.3.1.5 Checks on Guest Non-Register State 
The following checks are performed on fields in the guest-state area corresponding to non-register state: 
° Activity state. 


— The activity-state field must contain a value in the range 0 — 3, indicating an activity state supported by the 
implementation (see Section 23.4.2). Future processors may include support for other activity states. 
Software should read the VMX capability MSR IA32_VMX_MISC (see Appendix A.6) to determine what 
activity states are supported. 


— The activity State field must not indicate the HLT state if the DPL (bits 6:5) in the access-rights field for SS 
is not O. 


— The activity-state field must indicate the active state if the interruptibility-state field indicates blocking by 
either MOV-SS or by STI (if either bit O or bit 1 in that field is 1). 


— If the valid bit (bit 31) in the VM-entry interruption-information field is 1, the interruption to be delivered 
(as defined by interruption type and vector) must not be one that would normally be blocked while a logical 
processor is in the activity state corresponding to the contents of the activity-state field. The following 
items enumerate the interruptions (as specified in the VM-entry interruption-information field) whose 
injection is allowed for the different activity states: 


¢ Active. Any interruption is allowed. 
¢ HLT. The only events allowed are the following: 
— Those with interruption type external interrupt or non-maskable interrupt (NMI). 


— Those with interruption type hardware exception and vector 1 (debug exception) or vector 18 
(machine-check exception). 


— Those with interruption type other event and vector 0 (pending MTF VM exit). 


See Table 23-15 in Section 23.8.3 for details regarding the format of the VM-entry interruption- 
information field. 


¢ Shutdown. Only NMIs and machine-check exceptions are allowed. 
¢ Wait-for-SIPI. No interruptions are allowed. 


1. Software can determine the number N by executing CPUID with 8OOO0008H in EAX. The number of linear-address bits supported is 
returned in bits 15:8 of EAX. 


1. If the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PE must be 1 in VMX operation, bit 0 in the CRO field must be 1 
unless the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are both 1. 


2. As noted in Section 23.4.1, SS.DPL corresponds to the logical processor's current privilege level (CPL). 
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— The activity-state field must not indicate the wait-for-SIPI state if the “entry to SMM” VM-entry control is 1. 
* Interruptibility state. 

— The reserved bits (bits 31:5) must be 0. 

— The field cannot indicate blocking by both STI and MOV SS (bits 0 and 1 cannot both be 1). 

— Bit 0 (blocking by STI) must be 0 if the IF flag (bit 9) is 0 in the RFLAGS field. 


— Bit 0 (blocking by STI) and bit 1 (blocking by MOV-SS) must both be 0 if the valid bit (bit 31) in the 
VM-entry interruption-information field is 1 and the interruption type (bits 10:8) in that field has value 0, 
indicating external interrupt, or value 2, indicating non-maskable interrupt (NMI). 


— Bit 2 (blocking by SMI) must be 0 if the processor is not in SMM. 
— Bit 2 (blocking by SMI) must be 1 if the “entry to SMM” VM-entry control is 1. 


— Bit 3 (blocking by NMI) must be 0 if the “virtual NMIs” VM-execution control is 1, the valid bit (bit 31) in the 
VM-entry interruption-information field is 1, and the interruption type (bits 10:8) in that field has value 2 
(indicating NMI). 


— If bit 4 (enclave interruption) is 1, bit 1 (blocking by MOV-SS) must be 0 and the processor must support 
for SGX by enumerating CPUID.(EAX=07H,ECX=0):EBX.SGX[bit 2] as 1. 


NOTE 


If the “virtual NMIs” VM-execution control is 0, there is no requirement that bit 3 be 0 if the valid bit 
in the VM-entry interruption-information field is 1 and the interruption type in that field has value 2. 


¢ Pending debug exceptions. 


— Bits 11:4, bit 13, bit 15, and bits 63:17 (bits 31:17 on processors that do not support Intel 64 architecture) 
must be 0. 


— The following checks are performed if any of the following holds: (1) the interruptibility-state field indicates 
blocking by STI (bit O in that field is 1); (2) the interruptibility-state field indicates blocking by MOV SS 
(bit 1 in that field is 1); or (3) the activity-state field indicates HLT: 


¢ Bit 14 (BS) must be 1 if the TF flag (bit 8) in the RFLAGS field is 1 and the BTF flag (bit 1) in the 
IA32_DEBUGCTL field is 0. 


¢ Bit 14 (BS) must be 0 if the TF flag (bit 8) in the RFLAGS field is 0 or the BTF flag (bit 1) in the 
IA32_DEBUGCTL field is 1. 


— The following checks are performed if bit 16 (RTM) is 1: 


¢ Bits 11:0, bits 15:13, and bits 63:17 (bits 31:17 on processors that do not support Intel 64 archi- 
tecture) must be 0; bit 12 must be 1. 


¢ The processor must support for RTM by enumerating CPUID.(EAX=07H,ECX=0):EBX[bit 11] as 1. 
¢ The interruptibility-state field must not indicate blocking by MOV SS (bit 1 in that field must be 0). 
¢ VMCS link pointer. The following checks apply if the field contains a value other than FFFFFFFF_FFFFFFFFH: 
— Bits 11:0 must be 0. 
— Bits beyond the processor's physical-address width must be 0.1/2 


— The 4 bytes located in memory referenced by the value of the field (as a physical address) must satisfy the 
following: 


* Bits 30:0 must contain the processor’s VMCS revision identifier (see Section 23.2).° 


1. Software can determine a processor's physical-address width by executing CPUID with BOOO0008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


2. If 1A32_VMX_BASIC[48] is read as 1, this field must not set any bits in the range 63:32; see Appendix A.1. 


3. Earlier versions of this manual specified that the VMCS revision identifier was a 32-bit field. For all processors produced prior to this 
change, bit 31 of the VMCS revision identifier was 0. 
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* Bit 31 must contain the setting of the “VMCS shadowing” VM-execution control.! This implies that the 
referenced VMCS is a shadow VMCS (see Section 23.10) if and only if the “VMCS shadowing” VM- 
execution control is 1. 


— If the processor is not in SMM or the “entry to SMM” VM-entry control is 1, the field must not contain the 
current VMCS pointer. 


— If the processor is in SMM and the “entry to SMM” VM-entry control is 0, the field must differ from the 
executive-VMCS pointer. 


25.3.1.6 Checks on Guest Page-Directory-Pointer-Table Entries 


If CRO.PG =1, CR4.PAE = 1, and IA32_EFER.LME = 0, the logical processor uses PAE paging (see Section 4.4 in 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A).2 When PAE paging is in use, the 
physical address in CR3 references a table of page-directory-pointer-table entries (PDPTEs). A MOV to CR3 
when PAE paging is in use checks the validity of the PDPTEs. 


A VM entry is to a guest that uses PAE paging if (1) bit 31 (corresponding to CRO.PG) is set in the CRO field in the 
guest-state area; (2) bit 5 (corresponding to CR4.PAE) is set in the CR4 field; and (3) the “IA-32e mode guest” 
VM-entry control is 0. Such a VM entry checks the validity of the PDPTEs: 


e If the “enable EPT” VM-execution control is 0, VM entry checks the validity of the PDPTEs referenced by the CR3 
field in the guest-state area if either (1) PAE paging was not in use before the VM entry; or (2) the value of CR3 
is changing as a result of the VM entry. VM entry may check their validity even if neither (1) nor (2) hold.? 


¢ If the “enable EPT” VM-execution control is 1, VM entry checks the validity of the PDPTE fields in the guest- 
state area (see Section 23.4.2). 


A VM entry to a guest that does not use PAE paging does not check the validity of any PDPTEs. 


A VM entry that checks the validity of the PDPTEs uses the same checks that are used when CR3 is loaded with 
MOV to CR3 when PAE paging is in use.* If MOV to CR3 would cause a general-protection exception due to the 
PDPTEs that would be loaded (e.g., because a reserved bit is set), the VM entry fails. 


25.3.2 Loading Guest State 


Processor state is updated on VM entries in the following ways: 

¢ Some state is loaded from the guest-state area. 

¢ Some state is determined by VM-entry controls. 

¢ The page-directory pointers are loaded based on the values of certain control registers. 


This loading may be performed in any order and in parallel with the checking of VMCS contents (see Section 
25.3.1). 


The loading of guest state is detailed in Section 25.3.2.1 to Section 25.3.2.4. These sections reference VMCS fields 
that correspond to processor state. Unless otherwise stated, these references are to fields in the guest-state area. 


In addition to the state loading described in this section, VM entries may load MSRs from the VM-entry MSR-load 
area (see Section 25.4). This loading occurs only after the state loading described in this section and the checking 
of VMCS contents described in Section 25.3.1. 


1. “VMCS shadowing" is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based V/M-execution 
controls is 0, VM entry functions as if the “VMCS shadowing” VM-execution control were 0. See Section 23.6.2. 

2. On processors that support Intel 64 architecture, the physical-address extension may support more than 36 physical-address bits. 
Software can determine the number physical-address bits supported by executing CPUID with BOOOOOO8H in EAX. The physical- 
address width is returned in bits 7:0 of EAX. 

3. “Enable EPT” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution controls 
is 0, VM entry functions as if the “enable EPT” VM-execution control were O. See Section 23.6.2. 

4. This implies that (1) bits 11:9 in each PDPTE are ignored; and (2) if bit O (present) is clear in one of the PDPTEs, bits 63:1 of that 
PDPTE are ignored. 
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25.3.2.1 Loading Guest Control Registers, Debug Registers, and MSRs 


The following items describe how guest control registers, debug registers, and MSRs are loaded on VM entry: 


CRO is loaded from the CRO field with the exception of the following bits, which are never modified on VM entry: 
ET (bit 4); reserved bits 15:6, 17, and 28:19; NW (bit 29) and CD (bit 30).! The values of these bits in the CRO 
field are ignored. 


CR3 and CR4 are loaded from the CR3 field and the CR4 field, respectively. 


If the “load debug controls” VM-entry control is 1, DR7 is loaded from the DR7 field with the exception that 
bit 12 and bits 15:14 are always 0 and bit 10 is always 1. The values of these bits in the DR7 field are ignored. 


The first processors to support the virtual-machine extensions supported only the i-setting of the “load 
debug controls” VM-entry control and thus always loaded DR7 from the DR7 field. 


The following describes how certain MSRs are loaded using fields in the guest-state area: 


— Ifthe “load debug controls” VM-entry control is 1, the IA32_DEBUGCTL MSR is loaded from the 
IA32_DEBUGCTL field. The first processors to support the virtual-machine extensions supported only the 1- 
setting of this control and thus always loaded the IA32_DEBUGCTL MSR from the IA32_DEBUGCTL field. 


— The IA32_SYSENTER_CS MSR is loaded from the IA32_SYSENTER_CS field. Since this field has only 32 bits, 
bits 63:32 of the MSR are cleared to O. 


— The IA32_SYSENTER_ESP and IA32_SYSENTER_EIP MSRs are loaded from the IA32_SYSENTER_ESP field 
and the IA32_SYSENTER_EIP field, respectively. On processors that do not support Intel 64 architecture, 
these fields have only 32 bits; bits 63:32 of the MSRs are cleared to 0. 


— The following are performed on processors that support Intel 64 architecture: 


¢ The MSRs FS.base and GS.base are loaded from the base-address fields for FS and GS, respectively 
(see Section 25.3.2.2). 


¢ If the “load IA32_EFER” VM-entry control is 0, bits in the IA32_EFER MSR are modified as follows: 
— IA32_EFER.LMA is loaded with the setting of the “IA-32e mode guest” VM-entry control. 


— If CRO is being loaded so that CRO.PG = 1, IA32_EFER.LME is also loaded with the setting of the 
“IA-32e mode guest” VM-entry control. Otherwise, IA32_EFER.LME is unmodified. 


See below for the case in which the “load IA32_EFER” VM-entry control is 1 


— Ifthe “load IA32_PERF_GLOBAL_CTRL” VM-entry control is 1, the IA32_PERF_GLOBAL_CTRL MSR is loaded 
from the IA32_PERF_GLOBAL_CTRL field. 


— If the “load IA32_PAT” VM-entry control is 1, the IA32_PAT MSR is loaded from the IA32_PAT field. 
— Ifthe “load IA32_EFER” VM-entry control is 1, the IA32_EFER MSR is loaded from the IA32_EFER field. 


— Ifthe “load IA32_BNDCFGS” VM-entry control is 1, the IA32_BNDCFGS MSR is loaded from the 
IA32_BNDCFGS field. 


— Ifthe “load IA32_RTIT_CTL” VM-entry control is 1, the IA32_RTIT_CTL MSR is loaded from the 
IA32_RTIT_CTL field. 


— Ifthe “load CET” VM-entry control is 1, the IA32_S_CET and IA32_INTERRUPT_SSP_TABLE_ADDR MSRs 
are loaded from the IA32_S_CET field and the IA32_INTERRUPT_SSP_TABLE_ADDR field, respectively. On 
processors that do not support Intel 64 architecture, these fields have only 32 bits; bits 63:32 of the MSRs 
are cleared to 0. 


— Ifthe “load PKRS” VM-entry control is 1, the IA32_PKRS MSR is loaded from the IA32_PKRS field. 


With the exception of FS.base and GS.base, any of these MSRs is subsequently overwritten if it appears in the 
VM-entry MSR-load area. See Section 25.4. 


. Bits 15:6, bit 17, and bit 28:19 of CRO and CRO.ET are unchanged by executions of MOV to CRO. Bits 15:6, bit 17, and bit 28:19 of 


CRO are always O and CRO.ET is always 1. 

If the capability MSR 1[A32_VMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, VM entry must be loading CRO so 
that CRO.PG = 1 unless the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution con- 
trols are both 1. 
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The SMBASE register is unmodified by all VM entries except those that return from SMM. 


25.3.2.2 Loading Guest Segment Registers and Descriptor-Table Registers 
For each of CS, SS, DS, ES, FS, GS, TR, and LDTR, fields are loaded from the guest-state area as follows: 


The unusable bit is loaded from the access-rights field. This bit can never be set for TR (see Section 25.3.1.2). 
If it is set for one of the other registers, the following apply: 


— For each of CS, SS, DS, ES, FS, and GS, uses of the segment cause faults (general-protection exception or 
stack-fault exception) outside 64-bit mode, just as they would had the segment been loaded using a null 
selector. This bit does not cause accesses to fault in 64-bit mode. 


— If this bit is set for LDTR, uses of LDTR cause general-protection exceptions in all modes, just as they would 
had LDTR been loaded using a null selector. 


If this bit is clear for any of CS, SS, DS, ES, FS, GS, TR, and LDTR, a null selector value does not cause a fault 
(general-protection exception or stack-fault exception). 


TR. The selector, base, limit, and access-rights fields are loaded. 
cs. 


— The following fields are always loaded: selector, base address, limit, and (from the access-rights field) the 
L, D, and G bits. 


— For the other fields, the unusable bit of the access-rights field is consulted: 
¢ If the unusable bit is 0, all of the access-rights field is loaded. 
¢ Ifthe unusable bit is 1, the remainder of CS access rights are undefined after VM entry. 
SS, DS, ES, FS, GS, and LDTR. 
— The selector fields are loaded. 
— For the other fields, the unusable bit of the corresponding access-rights field is consulted: 


¢ If the unusable bit is 0, the base-address, limit, and access-rights fields are loaded. 


¢ If the unusable bit is 1, the base address, the segment limit, and the remainder of the access rights are 
undefined after VM entry with the following exceptions: 


— Bits 3:0 of the base address for SS are cleared to 0. 


— SS.DPL is always loaded from the SS access-rights field. This will be the current privilege level 
(CPL) after the VM entry completes. 


— SS.B is always set to 1. 


— The base addresses for FS and GS are loaded from the corresponding fields in the VMCS. On 
processors that support Intel 64 architecture, the values loaded for base addresses for FS and GS 
are also manifest in the FS.base and GS.base MSRs. 


— On processors that support Intel 64 architecture, the base address for LDTR is set to an undefined 
but canonical value. 


— On processors that support Intel 64 architecture, bits 63:32 of the base addresses for SS, DS, and 
ES are cleared to 0. 


GDTR and IDTR are loaded using the base and limit fields. 


25.3.2.3 Loading Guest RIP, RSP, RFLAGS, and SSP 

RSP, RIP, and RFLAGS are loaded from the RSP field, the RIP field, and the RFLAGS field, respectively. 
If the “load CET” VM-entry control is 1, SSP (shadow-stack pointer) is loaded from the SSP field. 

The following items regard the upper 32 bits of these fields on VM entries that are not to 64-bit mode: 


Bits 63:32 of RSP are undefined outside 64-bit mode. Thus, a logical processor may ignore the contents of 
bits 63:32 of the RSP field on VM entries that are not to 64-bit mode. 
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° As noted in Section 25.3.1.4, bits 63:32 of the RIP and RFLAGS fields must be 0 on VM entries that are not to 
64-bit mode. (The same is true for SSP for VM entries that are not to 64-bit mode when the “load CET” VM- 
entry control is 1.) 


25.3.2.4 Loading Page-Directory-Pointer-Table Entries 


As noted in Section 25.3.1.6, the logical processor uses PAE paging if CRO.PG = 1, CR4.PAE = 1, and 
IA32_EFER.LME = 0. AVM entry to a guest that uses PAE paging loads the PDPTEs into internal, non-architectural 
registers based on the setting of the “enable EPT” VM-execution control: 


¢ If the control is 0, the PDPTEs are loaded from the page-directory-pointer table referenced by the physical 
address in the value of CR3 being loaded by the VM entry (see Section 25.3.2.1). The values loaded are treated 
as physical addresses in VMX non-root operation. 


¢ If the control is 1, the PDPTEs are loaded from corresponding fields in the guest-state area (see Section 
23.4.2). The values loaded are treated as guest-physical addresses in VMX non-root operation. 


25.3.2.5 Updating Non-Register State 


Section 27.3 describes how the VMX architecture controls how a logical processor manages information in the TLBs 
and paging-structure caches. The following items detail how VM entries invalidate cached mappings: 


e If the “enable VPID” VM-execution control is 0, the logical processor invalidates linear mappings and combined 
mappings associated with VPID OOOOH (for all PCIDs); combined mappings for VPID OOOOH are invalidated for 
all EP4TA values (EP4TA is the value of bits 51:12 of EPTP). 


e VM entries are not required to invalidate any guest-physical mappings, nor are they required to invalidate any 
linear mappings or combined mappings if the “enable VPID” VM-execution control is 1. 


If the “virtual-interrupt delivery” VM-execution control is 1, VM entry loads the values of RVI and SVI from the 
guest interrupt-status field in the VMCS (see Section 23.4.2). After doing so, the logical processor first causes PPR 
virtualization (Section 28.1.3) and then evaluates pending virtual interrupts (Section 28.2.1). 

If a virtual interrupt is recognized, it may be delivered in VMX non-root operation immediately after VM entry 


(including any specified event injection) completes; see Section 25.7.5. See Section 28.2.2 for details regarding 
the delivery of virtual interrupts. 


25.3.3. Clearing Address-Range Monitoring 


The Intel 64 and IA-32 architectures allow software to monitor a specified address range using the MONITOR and 
MWAIT instructions. See Section 8.10.4 in the Inte!® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 3A. VM entries clear any address-range monitoring that may be in effect. 


25.4 LOADING MSRS 


VM entries may load MSRs from the VM-entry MSR-load area (see Section 23.8.2). Specifically each entry in that 
area (up to the number specified in the VM-entry MSR-load count) is processed in order by loading the MSR indexed 
by bits 31:0 with the contents of bits 127:64 as they would be written by WRMSR.? 


Processing of an entry fails in any of the following cases: 
° The value of bits 31:0 is either COO00100H (the IA32_FS_BASE MSR) or C0000101 (the IA32_GS_BASE MSR). 


¢ The value of bits 31:8 is OO0008H, meaning that the indexed MSR is one that allows access to an APIC register 
when the local APIC is in x2APIC mode. 


¢ The value of bits 31:0 indicates an MSR that can be written only in system-management mode (SMM) and the 
VM entry did not commence in SMM. (IA32_SMM_MONITOR_CTL is an MSR that can be written only in SMM.) 


1. Because attempts to modify the value of IA32_EFER.LMA by WRMSR are ignored, attempts to modify it using the VM-entry MSR- 
load area are also ignored. 
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° The value of bits 31:0 indicates an MSR that cannot be loaded on VM entries for model-specific reasons. A 
processor may prevent loading of certain MSRs even if they can normally be written by WRMSR. Such model- 
specific behavior is documented in Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 4. 


¢ =6Bits 63:32 are not all 0. 


¢ Anattempt to write bits 127:64 to the MSR indexed by bits 31:0 of the entry would cause a general-protection 
exception if executed via WRMSR with CPL = 0.! 


The VM entry fails if processing fails for any entry. The logical processor responds to such failures by loading state 
from the host-state area, as it would for a VM exit. See Section 25.8. 


If any MSR is being loaded in such a way that would architecturally require a TLB flush, the TLBs are updated so 
that, after VM entry, the logical processor will not use any translations that were cached before the transition. 


25.5 TRACE-ADDRESS PRE-TRANSLATION (TAPT) 


When the “Intel PT uses guest physical addresses” VM-execution control is 1, the addresses used by Intel PT are 
treated as guest-physical addresses, and these are translated to physical addresses using EPT. 


VM entry uses trace-address pre-translation (TAPT) to prevent buffered trace data from being lost due to an 
EPT violation; see Section 24.5.4.2. VM entry uses TAPT only if Intel PT will be enabled following VM entry 
(IA32_RTIT_CTL.TraceEn = 1) and only if the “Intel PT uses guest physical addresses” VM-execution control is 1 


As noted in Section 24.5.4, TAPT may cause a VM exit due to an EPT violation, EPT misconfiguration, page-modifi- 
cation log-full event, or APIC access. If such a VM exit occurs as a result of TAPT during VM entry, the VM exit oper- 
ates as if it had occurred in VMX non-root operation after the VM entry completed (in the guest context). 


If TAPT during VM entry causes a VM exit, the VM entry does not perform event injection (Section 25.6), even if the 
valid bit in the VM-entry interruption-information field is 1. Such VM exits save the contents of VM-entry interrup- 
tion-information and VM-entry exception error code fields into the IDT-vectoring information and IDT-vectoring 
error code fields, respectively. 


25.6 EVENT INJECTION 


If the valid bit in the VM-entry interruption-information field (see Section 23.8.3) is 1, VM entry causes an event to 
be delivered (or made pending) after all components of guest state have been loaded (including MSRs) and after 
the VM-execution control fields have been established. 


¢ If the interruption type in the field is 0 (external interrupt), 2 (non-maskable interrupt); 3 (hardware 
exception), 4 (software interrupt), 5 (privileged software exception), or 6 (software exception), the event is 
delivered as described in Section 25.6.1. 


¢ If the interruption type in the field is 7 (other event) and the vector field is 0, an MTF VM exit is pending after 
VM entry. See Section 25.6.2. 


25.6.1 Vectored-Event Injection 


VM entry delivers an injected vectored event within the guest context established by VM entry. This means that 
delivery occurs after all components of guest state have been loaded (including MSRs) and after the VM-execution 


1. If CRO.PG = 1, WRMSR to the IA32_EFER MSR causes a general-protection exception if it would modify the LME bit. If VM entry has 
established CRO.PG = 1, the IA32_EFER MSR should not be included in the VM-entry MSR-load area for the purpose of modifying the 
LME bit. 
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control fields have been established.! The event is delivered using the vector in that field to select a descriptor in 
the IDT. Since event injection occurs after loading IDTR from the guest-state area, this is the guest IDT. 


Section 25.6.1.1 provides details of vectored-event injection. In general, the event is delivered exactly as if it had 
been generated normally. 


If event delivery encounters a nested exception (for example, a general-protection exception because the vector 
indicates a descriptor beyond the IDT limit), the exception bitmap is consulted using the vector of that exception: 


e If the bit for the nested exception is 0, the nested exception is delivered normally. If the nested exception is 
benign, it is delivered through the IDT. If it is contributory or a page fault, a double fault may be generated, 
depending on the nature of the event whose delivery encountered the nested exception. See Chapter 6, 
“Interrupt 8—Double Fault Exception (#DF)” in Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 3A.2 


e If the bit for the nested exception is 1, a VM exit occurs. Section 25.6.1.2 details cases in which event injection 
causes a VM exit. 


25.6.1.1. Details of Vectored-Event Injection 


The event-injection process is controlled by the contents of the VM-entry interruption information field (format 
given in Table 23-15), the VM-entry exception error-code field, and the VM-entry instruction-length field. The 
following items provide details of the process: 


¢ The value pushed on the stack for RFLAGS is generally that which was loaded from the guest-state area. The 
value pushed for the RF flag is not modified based on the type of event being delivered. However, the pushed 
value of RFLAGS may be modified if a software interrupt is being injected into a guest that will be in virtual- 
8086 mode (see below). After RFLAGS is pushed on the stack, the value in the RFLAGS register is modified as 
is done normally when delivering an event through the IDT. 


¢ The instruction pointer that is pushed on the stack depends on the type of event and whether nested exceptions 
occur during its delivery. The term current guest RIP refers to the value to be loaded from the guest-state 
area. The value pushed is determined as follows: 


— If VM entry successfully injects (with no nested exception) an event with interruption type external 
interrupt, NMI, or hardware exception, the current guest RIP is pushed on the stack. 


— If VM entry successfully injects (with no nested exception) an event with interruption type software 
interrupt, privileged software exception, or software exception, the current guest RIP is incremented by the 
VM-entry instruction length before being pushed on the stack. 


— If VM entry encounters an exception while injecting an event and that exception does not cause a VM exit, 
the current guest RIP is pushed on the stack regardless of event type or VM-entry instruction length. If the 
encountered exception does cause a VM exit that saves RIP, the saved RIP is current guest RIP. 


° If the deliver-error-code bit (bit 11) is set in the VM-entry interruption-information field, the contents of the 
VM-entry exception error-code field is pushed on the stack as an error code would be pushed during delivery of 
an exception. 


° DR6, DR7, and the IA32_DEBUGCTL MSR are not modified by event injection, even if the event has vector 1 
(normal deliveries of debug exceptions, which have vector 1, do update these registers). 


° If VM entry is injecting a software interrupt and the guest will be in virtual-8086 mode (RFLAGS.VM = 1), no 
general-protection exception can occur due to RFLAGS.IOPL < 3. A VM monitor should check RFLAGS.IOPL 
before injecting such an event and, if desired, inject a general-protection exception instead of a software 
interrupt. 


1. This does not imply that injection of an exception or interrupt will cause a VM exit due to the settings of VM-execution control fields 
(such as the exception bitmap) that would cause a VM exit if the event had occurred in VMX non-root operation. In contrast, a nested 
exception encountered during event delivery may cause a VM exit; see Section 25.6.1.1. 

2. Hardware exceptions with the following unused vectors are considered benign: 15 and 21-31. A hardware exception with vector 20 
is considered benign unless the processor supports the 1-setting of the “EPT-violation #VE” VM-execution control; in that case, it 
has the same severity as page faults. 


3. While these items refer to RIP, the width of the value pushed (16 bits, 32 bits, or 64 bits) is determined normally. 
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If VM entry is injecting a software interrupt and the guest will be in virtual-8086 mode with virtual-8086 mode 
extensions (RFLAGS.VM = CR4.VME = 1), event delivery is subject to VME-based interrupt redirection based 
on the software interrupt redirection bitmap in the task-state segment (TSS) as follows: 


— If bit nin the bitmap is clear (where n is the number of the software interrupt), the interrupt is directed to 
an 8086 program interrupt handler: the processor uses a 16-bit interrupt-vector table (IVT) located at 
linear address zero. If the value of RFLAGS.IOPL is less than 3, the following modifications are made to the 
value of RFLAGS that is pushed on the stack: IOPL is set to 3, and IF is set to the value of VIF. 


— If bit nin the bitmap is set (where n is the number of the software interrupt), the interrupt is directed to a 
protected-mode interrupt handler. (In other words, the injection is treated as described in the next item.) 
In this case, the software interrupt does not invoke such a handler if RFLAGS.IOPL < 3 (a general- 
protection exception occurs instead). However, as noted above, RFLAGS.IOPL cannot cause an injected 
software interrupt to cause such a exception. Thus, in this case, the injection invokes a protected-mode 
interrupt handler independent of the value of RFLAGS.IOPL. 


Injection of events of other types are not subject to this redirection. 


If VM entry is injecting a software interrupt (not redirected as described above) or software exception, privilege 
checking is performed on the IDT descriptor being accessed as would be the case for executions of INT n, INT3, 
or INTO (the descriptor’s DPL cannot be less than CPL). There is no checking of RFLAGS.IOPL, even if the guest 
will be in virtual-8086 mode. Failure of this check may lead to a nested exception. Injection of an event with 
interruption type external interrupt, NMI, hardware exception, and privileged software exception, or with inter- 
ruption type software interrupt and being redirected as described above, do not perform these checks. 


If VM entry is injecting a non-maskable interrupt (NMI) and the “virtual NMIs” VM-execution control is 1, 
virtual-NMI blocking is in effect after VM entry. 

The transition causes a last-branch record to be logged if the LBR bit is set in the IA32_ DEBUGCTL MSR. This is 
true even for events such as debug exceptions, which normally clear the LBR bit before delivery. 

The last-exception record MSRs (LERs) may be updated based on the setting of the LBR bit in the 
IA32_DEBUGCTL MSR. Events such as debug exceptions, which normally clear the LBR bit before they are 
delivered, and therefore do not normally update the LERs, may do so as part of VM-entry event injection. 


If injection of an event encounters a nested exception, the value of the EXT bit (bit 0) in any error code for that 
nested exception is determined as follows: 


— If event being injected has interruption type external interrupt, NMI, hardware exception, or privileged 
software exception and encounters a nested exception (but does not produce a double fault), the error code 
for that exception sets the EXT bit. 


— If event being injected is a software interrupt or a software exception and encounters a nested exception, 
the error code for that exception clears the EXT bit. 


— If event delivery encounters a nested exception and delivery of that exception encounters another 
exception (but does not produce a double fault), the error code for that exception sets the EXT bit. 


— Ifa double fault is produced, the error code for the double fault is OOOOH (the EXT bit is clear). 


25.6.1.2 VM Exits During Event Injection 


An event being injected never causes a VM exit directly regardless of the settings of the VM-execution controls. For 
example, setting the “NMI exiting” VM-execution control to 1 does not cause a VM exit due to injection of an NMI. 


However, the event-delivery process may lead to a VM exit: 


If the vector in the VM-entry interruption-information field identifies a task gate in the IDT, the attempted task 
switch may cause a VM exit just as it would had the injected event occurred during normal execution in VMX 
non-root operation (see Section 24.4.2). 


If event delivery encounters a nested exception, a VM exit may occur depending on the contents of the 
exception bitmap (see Section 24.2). 


If event delivery generates a double-fault exception (due to a nested exception); the logical processor 
encounters another nested exception while attempting to call the double-fault handler; and that exception does 
not cause a VM exit due to the exception bitmap; then a VM exit occurs due to triple fault (see Section 24.2). 
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° If event delivery injects a double-fault exception and encounters a nested exception that does not cause a 
VM exit due to the exception bitmap, then a VM exit occurs due to triple fault (see Section 24.2). 


e If the “virtualize APIC accesses” VM-execution control is 1 and event delivery generates an access to the APIC- 
access page, that access is treated as described in Section 28.4 and may cause a VM exit.! 


If the event-delivery process does cause a VM exit, the processor state before the VM exit is determined just as it 
would be had the injected event occurred during normal execution in VMX non-root operation. If the injected event 
directly accesses a task gate that cause a VM exit or if the first nested exception encountered causes a VM exit, 
information about the injected event is saved in the IDT-vectoring information field (see Section 26.2.4). 


25.6.1.3 Event Injection for VM Entries to Real-Address Mode 


If VM entry is loading CRO.PE with 0, any injected vectored event is delivered as would normally be done in real- 
address mode.? Specifically, VM entry uses the vector provided in the VM-entry interruption-information field to 
select a 4-byte entry from an interrupt-vector table at the linear address in IDTR.base. Further details are provided 
in Section 15.1.4 in Volume 3A of the JA-32 Intel® Architecture Software Developer’s Manual. 


Because bit 11 (deliver error code) in the VM-entry interruption-information field must be 0 if CRO.PE will be 0 after 
VM entry (see Section 25.2.1.3), vectored events injected with CRO.PE = 0 do not push an error code on the stack. 
This is consistent with event delivery in real-address mode. 


If event delivery encounters a fault (due to a violation of IDTR. limit or of SS.limit), the fault is treated as if it had 
occurred during event delivery in VMX non-root operation. Such a fault may lead to a VM exit as discussed in 
Section 25.6.1.2. 


25.6.2 Injection of Pending MTF VM Exits 


If the interruption type in the VM-entry interruption-information field is 7 (other event) and the vector field is 0, 
VM entry causes an MTF VM exit to be pending on the instruction boundary following VM entry. This is the case 
even if the “monitor trap flag” VM-execution control is 0. See Section 24.5.2 for the treatment of pending MTF 
VM exits. 


25.7. SPECIAL FEATURES OF VM ENTRY 


This section details a variety of features of VM entry. It uses the following terminology: a VM entry is vectoring if 
the valid bit (bit 31) of the VM-entry interruption information field is 1 and the interruption type in the field is 0 
(external interrupt), 2 (non-maskable interrupt); 3 (hardware exception), 4 (software interrupt), 5 (privileged 
software exception), or 6 (software exception). 


25.7.1‘ Interruptibility State 

The interruptibility-state field in the guest-state area (see Table 23-3) contains bits that control blocking by STI, 

blocking by MOV SS, and blocking by NMI. This field impacts event blocking after VM entry as follows: 

¢ If the VM entry is vectoring, there is no blocking by STI or by MOV SS following the VM entry, regardless of the 
contents of the interruptibility-state field. 

° If the VM entry is not vectoring, the following apply: 


— Events are blocked by STI if and only if bit 0 in the interruptibility-state field is 1. This blocking is cleared 
after the guest executes one instruction or incurs an exception (including a debug exception made pending 
by VM entry; see Section 25.7.3). 


1. “Virtualize APIC accesses” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execu- 
tion controls is 0, VM entry functions as if the “virtualize APIC accesses” VM-execution control were 0. See Section 23.6.2. 

2. If the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PE must be 1 in VMX operation, VM entry must be loading CRO.PE 
with 1 unless the “unrestricted guest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are 
both 1. 
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— Events are blocked by MOV SS if and only if bit 1 in the interruptibility-state field is 1. This may affect the 
treatment of pending debug exceptions; see Section 25.7.3. This blocking is cleared after the guest 
executes one instruction or incurs an exception (including a debug exception made pending by VM entry). 


The blocking of non-maskable interrupts (NMIs) is determined as follows: 


— Ifthe “virtual NMIs” VM-execution control is 0, NMIs are blocked if and only if bit 3 (blocking by NMI) in the 
interruptibility-state field is 1. If the “NMI exiting” VM-execution control is 0, execution of the IRET 
instruction removes this blocking (even if the instruction generates a fault). If the “NMI exiting” control is 
1, IRET does not affect this blocking. 


— The following items describe the use of bit 3 (blocking by NMI) in the interruptibility-state field if the 
“virtual NMIs” VM-execution control is 1: 


¢ The bit’s value does not affect the blocking of NMIs after VM entry. NMIs are not blocked in VMX non- 
root operation (except for ordinary blocking for other reasons, such as by the MOV SS instruction, the 
wait-for-SIPI state, etc.) 


¢ The bit’s value determines whether there is virtual-NMI blocking after VM entry. If the bit is 1, virtual- 
NMI blocking is in effect after VM entry. If the bit is 0, there is no virtual-NMI blocking after VM entry 
unless the VM entry is injecting an NMI (see Section 25.6.1.1). Execution of IRET removes virtual-NMI 
blocking (even if the instruction generates a fault). 


If the “NMI exiting” VM-execution control is 0, the “virtual NMIs” control must be 0; see Section 25.2.1.1. 
Blocking of system-management interrupts (SMIs) is determined as follows: 


— Ifthe VM entry was not executed in system-management mode (SMM), SMI blocking is unchanged by 
VM entry. 


— Ifthe VM entry was executed in SMM, SMIs are blocked after VM entry if and only if the bit 2 in the inter- 
ruptibility-state field is 1. 


25.7.2 Activity State 


The activity-state field in the guest-state area controls whether, after VM entry, the logical processor is active or in 
one of the inactive states identified in Section 23.4.2. The use of this field is determined as follows: 


If the VM entry is vectoring, the logical processor is in the active state after VM entry. While the consistency 
checks described in Section 25.3.1.5 on the activity-state field do apply in this case, the contents of the 
activity-state field do not determine the activity state after VM entry. 


If the VM entry is not vectoring, the logical processor ends VM entry in the activity state specified in the guest- 
state area. If VM entry ends with the logical processor in an inactive activity state, the VM entry generates any 
special bus cycle that is normally generated when that activity state is entered from the active state. If 

VM entry would end with the logical processor in the shutdown state and the logical processor is in SMX 
operation,! an Intel® TXT shutdown condition occurs. The error code used is 0000H, indicating “legacy 
shutdown.” See Inte/® Trusted Execution Technology Preliminary Architecture Specification. 


Some activity states unconditionally block certain events. The following blocking is in effect after any VM entry 
that puts the processor in the indicated state: 


— The active state blocks start-up IPIs (SIPIs). SIPIs that arrive while a logical processor is in the active state 
and in VMX non-root operation are discarded and do not cause VM exits. 


— The HLT state blocks start-up IPIs (SIPIs). SIPIs that arrive while a logical processor is in the HLT state and 
in VMX non-root operation are discarded and do not cause VM exits. 


— The shutdown state blocks external interrupts and SIPIs. External interrupts that arrive while a logical 
processor is in the shutdown state and in VMX non-root operation do not cause VM exits even if the 
“external-interrupt exiting” VM-execution control is 1. SIPIs that arrive while a logical processor is in the 
shutdown state and in VMX non-root operation are discarded and do not cause VM exits. 


1. 


A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. See 
Chapter 6, “Safer Mode Extensions Reference,” in Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 2B. 
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— The wait-for-SIPI state blocks external interrupts, non-maskable interrupts (NMIs), INIT signals, and 
system-management interrupts (SMIs). Such events do not cause VM exits if they arrive while a logical 
processor is in the wait-for-SIPI state and in VMX non-root operation. 


25.7.3 Delivery of Pending Debug Exceptions after VM Entry 


The pending debug exceptions field in the guest-state area indicates whether there are debug exceptions that have 
not yet been delivered (see Section 23.4.2). This section describes how these are treated on VM entry. 


There are no pending debug exceptions after VM entry if any of the following are true: 


¢ The VM entry is vectoring with one of the following interruption types: external interrupt, non-maskable 
interrupt (NMI), hardware exception, or privileged software exception. 


¢ = The interruptibility-state field does not indicate blocking by MOV SS and the VM entry is vectoring with either of 
the following interruption type: software interrupt or software exception. 


¢ The VM entry is not vectoring and the activity-state field indicates either shutdown or wait-for-SIPI. 


If none of the above hold, the pending debug exceptions field specifies the debug exceptions that are pending for 
the guest. There are valid pending debug exceptions if either the BS bit (bit 14) or the enable-breakpoint bit 
(bit 12) is 1. If there are valid pending debug exceptions, they are handled as follows: 


¢ If the VM entry is not vectoring, the pending debug exceptions are treated as they would had they been 
encountered normally in guest execution: 


— Ifthe logical processor is not blocking such exceptions (the interruptibility-state field indicates no blocking 
by MOV SS), a debug exception is delivered after VM entry (see below). 


— Ifthe logical processor is blocking such exceptions (due to blocking by MOV SS), the pending debug 
exceptions are held pending or lost as would normally be the case. 


¢ If the VM entry is vectoring (with interruption type software interrupt or software exception and with blocking 
by MOV SS), the following items apply: 


— For injection of a software interrupt or of a software exception with vector 3 (#BP) or vector 4 (#OF) — or 
a privileged software exception with vector 1 (#DB) — the pending debug exceptions are treated as they 
would had they been encountered normally in guest execution if the corresponding instruction (INT1, INT3, 
or INTO) were executed after a MOV SS that encountered a debug trap. 


— For injection of a software exception with a vector other than 3 and 4, the pending debug exceptions may 
be lost or they may be delivered after injection (see below). 


If there are no valid pending debug exceptions (as defined above), no pending debug exceptions are delivered after 
VM entry. 


If a pending debug exception is delivered after VM entry, it has the priority of “traps on the previous instruction” 
(see Section 6.9 in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). Thus, INIT 
signals and system-management interrupts (SMIs) take priority of such an exception, as do VM exits induced by 
the TPR threshold (see Section 25.7.7) and pending MTF VM exits (see Section 25.7.8. The exception takes priority 
over any pending non-maskable interrupt (NMI) or external interrupt and also over VM exits due to the 1-settings 
of the “interrupt-window exiting” and “NMI-window exiting” VM-execution controls. 


A pending debug exception delivered after VM entry causes a VM exit if the bit 1 (#DB) is 1 in the exception 
bitmap. If it does not cause a VM exit, it updates DR6 normally. 


25.7.4 VMX-Preemption Timer 


If the “activate VMX-preemption timer” VM-execution control is 1, VM entry starts the VMX-preemption timer with 
the unsigned value in the VMX-preemption timer-value field. 


It is possible for the VMX-preemption timer to expire during VM entry (e.g., if the value in the VMX-preemption 
timer-value field is zero). If this happens (and if the VM entry was not to the wait-for-SIPI state), a VM exit occurs 
with its normal priority after any event injection and before execution of any instruction following VM entry. For 
example, any pending debug exceptions established by VM entry (see Section 25.7.3) take priority over a timer- 


25-24 Vol. 3C 


VM ENTRIES 


induced VM exit. (The timer-induced VM exit will occur after delivery of the debug exception, unless that exception 
or its delivery causes a different VM exit.) 


See Section 24.5.1 for details of the operation of the VMX-preemption timer in VMX non-root operation, including 
the blocking and priority of the VM exits that it causes. 


25.7.5 —_Interrupt-Window Exiting and Virtual-Interrupt Delivery 


If “interrupt-window exiting” VM-execution control is 1, an open interrupt window may cause a VM exit immedi- 
ately after VM entry (see Section 24.2 for details). If the “interrupt-window exiting” VM-execution control is 0 but 
the “virtual-interrupt delivery” VM-execution control is 1, a virtual interrupt may be delivered immediately after 
VM entry (see Section 25.3.2.5 and Section 28.2.1). 


The following items detail the treatment of these events: 
¢ These events occur after any event injection specified for VM entry. 


¢ Non-maskable interrupts (NMIs) and higher priority events take priority over these events. These events take 
priority over external interrupts and lower priority events. 


° These events wake the logical processor if it just entered the HLT state because of a VM entry (see Section 
25.7.2). They do not occur if the logical processor just entered the shutdown state or the wait-for-SIPI state. 


25.7.6 NMI-Window Exiting 

The “NMI-window exiting” VM-execution control may cause a VM exit to occur immediately after VM entry (see 
Section 24.2 for details). 

The following items detail the treatment of these VM exits: 

¢ These VM exits follow event injection if such injection is specified for VM entry. 


° Debug-trap exceptions (see Section 25.7.3) and higher priority events take priority over VM exits caused by 
this control. VM exits caused by this control take priority over non-maskable interrupts (NMIs) and lower 
priority events. 


¢ VM exits caused by this control wake the logical processor if it just entered either the HLT state or the shutdown 
state because of a VM entry (see Section 25.7.2). They do not occur if the logical processor just entered the 
wait-for-SIPI state. 


25.7.7. \/M Exits Induced by the TPR Threshold 


If the “use TPR shadow” and “virtualize APIC accesses” VM-execution controls are both 1 and the “virtual-interrupt 
delivery” VM-execution control is 0, a VM exit occurs immediately after VM entry if the value of bits 3:0 of the TPR 
threshold VM-execution control field is greater than the value of bits 7:4 of VTPR (see Section 28.1.1).! 


The following items detail the treatment of these VM exits: 


° The VM exits are not blocked if RFLAGS.IF = 0 or by the setting of bits in the interruptibility-state field in guest- 
state area. 


¢ The VM exits follow event injection if such injection is specified for VM entry. 


¢ VM exits caused by this control take priority over system-management interrupts (SMIs), INIT signals, and 
lower priority events. They thus have priority over the VM exits described in Section 25.7.5, Section 25.7.6, 
and Section 25.7.8, as well as any interrupts or debug exceptions that may be pending at the time of VM entry. 


¢ These VM exits wake the logical processor if it just entered the HLT state as part of a VM entry (see Section 
25.7.2). They do not occur if the logical processor just entered the shutdown state or the wait-for-SIPI state. 


1. “Virtualize APIC accesses” and “virtual-interrupt delivery” are secondary processor-based VM-execution controls. If bit 31 of the pri- 
mary processor-based VM-execution controls is 0, VM entry functions as if these controls were O. See Section 23.6.2. 
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If such a VM exit is suppressed because the processor just entered the shutdown state, it occurs after the 
delivery of any event that cause the logical processor to leave the shutdown state while remaining in VMX 
non-root operation (e.g., due to an NMI that occurs while the “NMI-exiting” VM-execution control is 0). 


° The basic exit reason is “TPR below threshold.” 


25.7.8 Pending MTF VM Exits 


As noted in Section 25.6.2, VM entry may cause an MTF VM exit to be pending immediately after VM entry. The 
following items detail the treatment of these VM exits: 


¢ System-management interrupts (SMIs), INIT signals, and higher priority events take priority over these 
VM exits. These VM exits take priority over debug-trap exceptions and lower priority events. 


¢ These VM exits wake the logical processor if it just entered the HLT state because of a VM entry (see Section 
25.7.2). They do not occur if the logical processor just entered the shutdown state or the wait-for-SIPI state. 


25.7.9 VM Entries and Advanced Debugging Features 


VM entries are not logged with last-branch records, do not produce branch-trace messages, and do not update the 
branch-trace store. 


25.8 VM-ENTRY FAILURES DURING OR AFTER LOADING GUEST STATE 


VM-entry failures due to the checks identified in Section 25.3.1 and failures during the MSR loading identified in 
Section 25.4 are treated differently from those that occur earlier in VM entry. In these cases, the following steps 
take place: 


1. Information about the VM-entry failure is recorded in the VM-exit information fields: 
— Exit reason. 


¢ Bits 15:0 of this field contain the basic exit reason. It is loaded with a number indicating the general 
cause of the VM-entry failure. The following numbers are used: 


33. VM-entry failure due to invalid guest state. A VM entry failed one of the checks identified in Section 
25.3.1. 


34. VM-entry failure due to MSR loading. A VM entry failed in an attempt to load MSRs (see Section 
25.4). 


41. VM-entry failure due to machine-check event. A machine-check event occurred during VM entry 
(see Section 25.9). 


¢ Bit 31 is set to 1 to indicate a VM-entry failure. 
¢ The remainder of the field (bits 30:16) is cleared. 
— Exit qualification. This field is set based on the exit reason. 


¢ VM-entry failure due to invalid guest state. In most cases, the exit qualification is cleared to 0. The 
following non-zero values are used in the cases indicated: 


1. Not used. 
2. Failure was due to a problem loading the PDPTEs (see Section 25.3.1.6). 


3. Failure was due to an attempt to inject a non-maskable interrupt (NMI) into a guest that is blocking 
events through the STI blocking bit in the interruptibility-state field. 


4. Failure was due to an invalid VMCS link pointer (see Section 25.3.1.5). 
VM-entry checks on guest-state fields may be performed in any order. Thus, an indication by exit 


qualification of one cause does not Maal that there are not also other errors. Different processors 
may give different exit qualifications for the same VMCS. 
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¢ \VM-entry failure due to MSR loading. The exit qualification is loaded to indicate which entry in the 
VM-entry MSR-load area caused the problem (1 for the first entry, 2 for the second, etc.). 


— All other VM-exit information fields are unmodified. 


Processor state is loaded as would be done on a VM exit (see Section 26.5). If this results in 
[CR4.PAE & CRO.PG & ~IA32_EFER.LMA] = 1, page-directory-pointer-table entries (PDPTEs) may be checked 
and loaded (see Section 26.5.4). 


The state of blocking by NMI is what it was before VM entry. 
MSRs are loaded as specified in the VM-exit MSR-load area (see Section 26.6). 


Although this process resembles that of a VM exit, many steps taken during a VM exit do not occur for these 
VM-entry failures: 


Most VM-exit information fields are not updated (see step 1 above). 

The valid bit in the VM-entry interruption-information field is not cleared. 
The guest-state area is not modified. 

No MSRs are saved into the VM-exit MSR-store area. 


25.9 MACHINE-CHECK EVENTS DURING VM ENTRY 


If a machine-check event occurs during a VM entry, one of the following occurs: 


The machine-check event is handled as if it occurred before the VM entry: 


— If CR4.MCE = 0, operation of the logical processor depends on whether the logical processor is in SMX 
operation: 


* If the logical processor is in SMX operation, an Intel® TXT shutdown condition occurs. The error code 
used is OOOCH, indicating “unrecoverable machine-check condition.” 


¢ Ifthe logical processor is outside SMX operation, it goes to the shutdown state. 
— If CR4.MCE = 1, a machine-check exception (#MC) is delivered through the IDT. 
The machine-check event is handled after VM entry completes: 


— Ifthe VM entry ends with CR4.MCE = 0, operation of the logical processor depends on whether the logical 
processor is in SMX operation: 


* If the logical processor is in SMX operation, an Intel® TXT shutdown condition occurs with error code 
OQOOCH (unrecoverable machine-check condition). 


¢ Ifthe logical processor is outside SMX operation, it goes to the shutdown state. 

— Ifthe VM entry ends with CR4.MCE = 1, a machine-check exception (#MC) is generated: 
¢ If bit 18 (4MC) of the exception bitmap is 0, the exception is delivered through the guest IDT. 
¢ If bit 18 of the exception bitmap is 1, the exception causes a VM exit. 


AVM-entry failure occurs as described in Section 25.8. The basic exit reason is 41, for “VM-entry failure due to 
machine-check event.” 


The first option is not used if the machine-check event occurs after any guest state has been loaded. The second 
option is used only if VM entry is able to load all guest state. 


A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. A logi- 
cal processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the last 
execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference,” in Intel® 64 and IA-32 Architectures Software 
Developer's Manual, Volume 2B. 
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VM exits occur in response to certain instructions and events in VMX non-root operation as detailed in Section 24.1 
through Section 24.2. VM exits perform the following operations: 


1. Information about the cause of the VM exit is recorded in the VM-exit information fields and VM-entry control 
fields are modified as described in Section 26.2. 


Processor state is saved in the guest-state area (Section 26.3). 


MSRs may be saved in the VM-exit MSR-store area (Section 26.4). This step is not performed for SMM VM exits 
that activate the dual-monitor treatment of SMIs and SMM. 


4. The following may be performed in parallel and in any order (Section 26.5): 


— Processor state is loaded based in part on the host-state area and some VM-exit controls. This step is not 
performed for SMM VM exits that activate the dual-monitor treatment of SMIs and SMM. See Section 
30.15.6 for information on how processor state is loaded by such VM exits. 


— Address-range monitoring is cleared. 


5. MSRs may be loaded from the VM-exit MSR-load area (Section 26.6). This step is not performed for SMM 
VM exits that activate the dual-monitor treatment of SMIs and SMM. 


VM exits are not logged with last-branch records, do not produce branch-trace messages, and do not update the 
branch-trace store. 


Section 26.1 clarifies the nature of the architectural state before a VM exit begins. The steps described above are 
detailed in Section 26.2 through Section 26.6. 


Section 30.15 describes the dual-monitor treatment of system-management interrupts (SMIs) and system- 
management mode (SMM). Under this treatment, ordinary transitions to SMM are replaced by VM exits to a sepa- 
rate SMM monitor. Called SMM VM exits, these are caused by the arrival of an SMI or the execution of VMCALL in 
VMX root operation. SMM VM exits differ from other VM exits in ways that are detailed in Section 30.15.2. 


26.1 ARCHITECTURAL STATE BEFORE A VM EXIT 


This section describes the architectural state that exists before a VM exit, especially for VM exits caused by events 
that would normally be delivered through the IDT. Note the following: 


¢ Anexception causes a VM exit directly if the bit corresponding to that exception is set in the exception bitmap. 
A non-maskable interrupt (NMI) causes a VM exit directly if the “NMI exiting” VM-execution control is 1. An 
external interrupt causes a VM exit directly if the “external-interrupt exiting” VM-execution control is 1. A start- 
up IPI (SIPI) that arrives while a logical processor is in the wait-for-SIPI activity state causes a VM exit directly. 
INIT signals that arrive while the processor is not in the wait-for-SIPI activity state cause VM exits directly. 


¢ An exception, NMI, external interrupt, or software interrupt causes a VM exit indirectly if it does not do so 
directly but delivery of the event causes a nested exception, double fault, task switch, APIC access (see Section 
28.4), EPT violation, EPT misconfiguration, page-modification log-full event (see Section 27.2.6), or SPP- 
related event (see Section 27.2.4) that causes a VM exit. 


¢ Anevent results in a VM exit if it causes a VM exit (directly or indirectly). 
The following bullets detail when architectural state is and is not updated in response to VM exits: 


¢ If an event causes a VM exit directly, it does not update architectural state as it would have if it had it not 
caused the VM exit: 


— Adebug exception does not update DR6, DR7, or IA32_DEBUGCTL. (Information about the nature of the 
debug exception is saved in the exit qualification field.) 


— A page fault does not update CR2. (The linear address causing the page fault is saved in the exit-qualifi- 
cation field.) 
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— An NMI causes subsequent NMIs to be blocked, but only after the VM exit completes. 


— Anexternal interrupt does not acknowledge the interrupt controller and the interrupt remains pending, 
unless the “acknowledge interrupt on exit” VM-exit control is 1. In such a case, the interrupt controller is 
acknowledged and the interrupt is no longer pending. 


— The flags LO - L3 in DR7 (bit 0, bit 2, bit 4, and bit 6) are not cleared when a task switch causes a VM exit. 


— Ifatask switch causes a VM exit, none of the following are modified by the task switch: old task-state 
segment (TSS); new TSS; old TSS descriptor; new TSS descriptor; RFLAGS.NT!; or the TR register. 


— No last-exception record is made if the event that would do so directly causes a VM exit. 


— Ifamachine-check exception causes a VM exit directly, this does not prevent machine-check MSRs from 
being updated. These are updated by the machine-check event itself and not the resulting machine-check 
exception. 


— Ifthe logical processor is in an inactive state (see Section 23.4.2) and not executing instructions, some 
events may be blocked but others may return the logical processor to the active state. Unblocked events 
may cause VM exits.2 If an unblocked event causes a VM exit directly, a return to the active state occurs 
only after the VM exit completes.? The VM exit generates any special bus cycle that is normally generated 
when the active state is entered from that activity state. 


MTF VM exits (see Section 24.5.2 and Section 25.7.8) are not blocked in the HLT activity state. If an MTF 
VM exit occurs in the HLT activity state, the logical processor returns to the active state only after the 
VM exit completes. MTF VM exits are blocked the shutdown state and the wait-for-SIPI state. 


e If an event causes a VM exit indirectly, the event does update architectural state: 


— Adebug exception updates DR6, DR7, and the IA32_DEBUGCTL MSR. No debug exceptions are considered 
pending. 


— A page fault updates CR2. 
— An NMI causes subsequent NMIs to be blocked before the VM exit commences. 
— Anexternal interrupt acknowledges the interrupt controller and the interrupt is no longer pending. 


— Ifthe logical processor had been in an inactive state, it enters the active state and, before the VM exit 
commences, generates any special bus cycle that is normally generated when the active state is entered 
from that activity state. 


— There is no blocking by STI or by MOV SS when the VM exit commences. 


— Processor state that is normally updated as part of delivery through the IDT (CS, RIP, SS, RSP, RFLAGS) is 
not modified. However, the incomplete delivery of the event may write to the stack. 


— The treatment of last-exception records is implementation dependent: 


* Some processors make a last-exception record when beginning the delivery of an event through the IDT 
(before it can encounter a nested exception). Such processors perform this update even if the event 
encounters a nested exception that causes a VM exit (including the case where nested exceptions lead 
to a triple fault). 


¢ Other processors delay making a last-exception record until event delivery has reached some event 
handler successfully (perhaps after one or more nested exceptions). Such processors do not update the 
last-exception record if a VM exit or triple fault occurs before an event handler is reached. 


1. This chapter uses the notation RAX, RIP, RSP, RFLAGS, etc. for processor registers because most processors that support VMX oper- 
ation also support Intel 64 architecture. For processors that do not support Intel 64 architecture, this notation refers to the 32-bit 
forms of those registers (EAX, EIP, ESP, EFLAGS, etc.). In a few places, notation such as EAX is used to refer specifically to lower 32 
bits of the indicated register. 


2. Ifa\VM exit takes the processor from an inactive state resulting from execution of a specific instruction (HLT or MWAIT), the value 
saved for RIP by that VM exit will reference the following instruction. 


3. Anexception is made if the logical processor had been inactive due to execution of MWAIT; in this case, it is considered to have 
become active before the VM exit. 
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¢ If the “virtual NMIs” VM-execution control is 1, VM entry injects an NMI, and delivery of the NMI causes a 
nested exception, double fault, task switch, EPT violation, EPT misconfiguration, page-modification log-full 
event, or SPP-related event, or APIC access that causes a VM exit, virtual-NMI blocking is in effect before the 
VM exit commences. 


° IfaVM exit results from a fault, EPT violation, EPT misconfiguration, page-modification log-full event, or SPP- 
related event that is encountered during execution of IRET and the “NMI exiting” VM-execution control is 0, any 
blocking by NMI is cleared before the VM exit commences. However, the previous state of blocking by NMI may 
be recorded in the exit qualification or in the VM-exit interruption-information field; see Section 26.2.3. 


° IfaVM exit results from a fault, EPT violation, EPT misconfiguration, page-modification log-full event, or SPP- 
related event that is encountered during execution of IRET and the “virtual NMIs” VM-execution control is 1, 
virtual-NMI blocking is cleared before the VM exit commences. However, the previous state of blocking by NMI 
may be recorded in the exit qualification or in the VM-exit interruption-information field; see Section 26.2.3. 


* Suppose that a VM exit is caused directly by an x87 FPU Floating-Point Error (#MF) or by any of the following 
events if the event was unblocked due to (and given priority over) an x87 FPU Floating-Point Error: an INIT 
signal, an external interrupt, an NMI, an SMI; or a machine-check exception. In these cases, there is no 
blocking by STI or by MOV SS when the VM exit commences. 

¢ Normally, a last-branch record may be made when an event is delivered through the IDT. However, if such an 
event results in a VM exit before delivery is complete, no last-branch record is made. 

¢ If machine-check exception results in a VM exit, processor state is suspect and may result in Suspect state 
being saved to the guest-state area. A VM monitor should consult the RIPV and EIPV bits in the 
IA32_MCG_STATUS MSR before resuming a guest that caused a VM exit resulting from a machine-check 
exception. 

° If aVM exit results from a fault, APIC access (see Section 28.4), EPT violation, EPT misconfiguration, page- 
modification log-full event, or SPP-related event that is encountered while executing an instruction, data 
breakpoints due to that instruction may have been recognized and information about them may be saved in the 
pending debug exceptions field (unless the VM exit clears that field; see Section 26.3.4). 


¢ The following VM exits are considered to happen after an instruction is executed: 
— VM exits resulting from debug traps (single-step, I/O breakpoints, and data breakpoints). 


— VM exits resulting from debug exceptions (data breakpoints) whose recognition was delayed by blocking by 
MOV SS. 
— VM exits resulting from some machine-check exceptions. 


— Trap-like VM exits due to execution of MOV to CR8 when the “CR8-load exiting” VM-execution control is 0 
and the “use TPR shadow” VM-execution control is 1 (see Section 28.3). (Such VM exits can occur only from 
64-bit mode and thus only on processors that support Intel 64 architecture.) 


— Trap-like VM exits due to execution of WRMSR when the “use MSR bitmaps” VM-execution control is 1; the 
value of ECX is in the range 800H-8FFH; and the bit corresponding to the ECX value in write bitmap for low 
MSRs is O; and the “virtualize x2APIC mode” VM-execution control is 1. See Section 28.5. 


— VM exits caused by APIC-write emulation (see Section 28.4.3.2) that result from APIC accesses as part of 
instruction execution. 


For these VM exits, the instruction’s modifications to architectural state complete before the VM exit occurs. 
Such modifications include those to the logical processor's interruptibility state (see Table 23-3). If there had 
been blocking by MOV SS, POP SS, or STI before the instruction executed, such blocking is no longer in effect. 


AVM exit that occurs in enclave mode sets bit 27 of the exit-reason field and bit 4 of the guest interruptibility-state 
field. Before such a VM exit is delivered, an Asynchronous Enclave Exit (AEX) occurs (see Chapter 35, “Enclave 
Exiting Events”). An AEX modifies architectural state (Section 35.3). In particular, the processor establishes the 
following architectural state as indicated: 


° The following bits in RFLAGS are cleared: CF, PF, AF, ZF, SF, OF, and RF. 

e FS and GS are restored to the values they had prior to the most recent enclave entry. 
¢ RIP is loaded with the AEP of interrupted enclave thread. 

* RSP is loaded from the URSP field in the enclave’s state-save area (SSA). 
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26.2 RECORDING VM-EXIT INFORMATION AND UPDATING VM-ENTRY CONTROL 
FIELDS 


VM exits begin by recording information about the nature of and reason for the VM exit in the VM-exit information 
fields. Section 26.2.1 to Section 26.2.5 detail the use of these fields. 


In addition to updating the VM-exit information fields, the valid bit (bit 31) is cleared in the VM-entry interruption- 
information field. If bit 5 of the IA32_VMX_MISC MSR (index 485H) is read as 1 (See Appendix A.6), the value of 
IA32_EFER.LMA is stored into the “IA-32e mode guest” VM-entry control.? 


26.2.1 Basic VM-Exit Information 


Section 23.9.1 defines the basic VM-exit information fields. The following items detail their use. 
¢ Exit reason. 


— Bits 15:0 of this field contain the basic exit reason. It is loaded with a number indicating the general cause 
of the VM exit. Appendix C lists the numbers used and their meaning. 


— Bit 27 of this field is set to 1 if the VM exit occurred while the logical processor was in enclave mode. 


Such VM exits include those caused by interrupts, non-maskable interrupts, system-management 
interrupts, INIT signals, and exceptions occurring in enclave mode as well as exceptions encountered 
during the delivery of such events incident to enclave mode. 


AVM exit also sets this bit if it is incident to delivery of an event injected by VM entry and the guest inter- 
ruptibility-state field indicates an enclave interruption (bit 4 of the field is 1). 


— The remainder of the field (bits 31:28 and bits 26:16) is cleared to 0 (certain SMM VM exits may set some 
of these bits; see Section 30.15.2.3).2 


° Exit qualification. This field is saved for VM exits due to the following causes: debug exceptions; page-fault 
exceptions; start-up IPIs (SIPIs); system-management interrupts (SMIs) that arrive immediately after the 
execution of I/O instructions; task switches; INVEPT; INVLPG; INVPCID; INVVPID; LGDT; LIDT; LLDT; LTR; 
SGDT; SIDT; SLDT; STR; VMCLEAR; VMPTRLD; VMPTRST; VMREAD; VMWRITE; VMXON; WBINVD; 
WBNOINVD; XRSTORS; XSAVES; control-register accesses; MOV DR; I/O instructions; MWAIT; accesses to the 
APIC-access page (see Section 28.4); EPT violations (see Section 27.2.3.2); EOI virtualization (see Section 
28.1.4); APIC-write emulation (see Section 28.4.3.3); page-modification log full (see Section 27.2.6); and 
SPP-related events (see Section 27.2.4). For all other VM exits, this field is cleared. The following items provide 
details: 


— Fora debug exception, the exit qualification contains information about the debug exception. The 
information has the format given in Table 26-1. 


Table 26-1. Exit Qualification for Debug Exceptions 


Bit Position(s) Contents 

3:0 B3 - BO. When set, each of these bits indicates that the corresponding breakpoint condition was met. Any of 
these bits may be set even if its corresponding enabling bit in DR7 is not set. 

12:4 Not currently defined. 

13 BD. When set, this bit indicates that the cause of the debug exception is “debug register access detected.” 

14 BS. When set, this bit indicates that the cause of the debug exception is either the execution of a single 


instruction (if RFLAGS.TF = 1 and [A32_DEBUGCTL.BTF = 0) or a taken branch (if 
RFLAGS.TF = DEBUGCTL.BTF = 1). 


1. Bit 5 of the IA32_VMX_MISC MSR is read as 1 on any logical processor that supports the 1-setting of the “unrestricted quest” VM- 
execution control. 


2. Bit 31 of this field is set on certain VM-entry failures; see Section 25.8. 
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Table 26-1. Exit Qualification for Debug Exceptions (Contd.) 


Bit Position(s) Contents 

15 Not currently defined. 

16 RTM. When set, this bit indicates that a debug exception (#DB) or a breakpoint exception (#BP) occurred 
inside an RTM region while advanced debugging of RTM transactional regions was enabled (see Section 
16.3.7, "RTM-Enabled Debugger Support,” of the Intel® 64 and IA-32 Architectures Software Developer's 
Manual, Volume 1).! 

63:17 Not currently defined. Bits 63:32 exist only on processors that support Intel 64 architecture. 

NOTES: 


1. In general, the format of this field matches that of DR6. However, DR6 clears bit 16 to indicate an RTM-related exception, while this 
field sets the bit to indicate that condition. 


For a page-fault exception, the exit qualification contains the linear address that caused the page fault. On 
processors that support Intel 64 architecture, bits 63:32 are cleared if the logical processor was not in 64- 
bit mode before the VM exit. 


If the page-fault exception occurred during execution of an instruction in enclave mode (and not during 
delivery of an event incident to enclave mode), bits 11:0 of the exit qualification are cleared. 


For a start-up IPI (SIPI), the exit qualification contains the SIPI vector information in bits 7:0. Bits 63:8 of 
the exit qualification are cleared to 0. 


For a task switch, the exit qualification contains details about the task switch, encoded as shown in 
Table 26-2. 


For INVLPG, the exit qualification contains the linear-address operand of the instruction. 


¢ On processors that support Intel 64 architecture, bits 63:32 are cleared if the logical processor was not 
in 64-bit mode before the VM exit. 


¢ Ifthe INVLPG source operand specifies an unusable segment, the linear address specified in the exit 
qualification will match the linear address that the INVLPG would have used if no VM exit occurred. This 
address is not architecturally defined and may be implementation-specific. 


Table 26-2. Exit Qualification for Task Switches 


Bit Position(s) Contents 
15:0 Selector of task-state segment (TSS) to which the guest attempted to switch 
29:16 Not currently defined 
31:30 Source of task switch initiation: 
0: CALL instruction 
1: IRET instruction 
2: JMP instruction 
3: Task gate in IDT 
63:32 Not currently defined. These bits exist only on processors that support Intel 64 architecture. 


For INVEPT, INVPCID, INVVPID, LGDT, LIDT, LLDT, LTR, SGDT, SIDT, SLDT, STR, VMCLEAR, VMPTRLD, 
VMPTRST, VMREAD, VMWRITE, VMXON, XRSTORS, and XSAVES, the exit qualification receives the value of 
the instruction’s displacement field, which is sign-extended to 64 bits if necessary (32 bits on processors 
that do not support Intel 64 architecture). If the instruction has no displacement (for example, has a 
register operand), zero is stored into the exit qualification. 


On processors that support Intel 64 architecture, an exception is made for RIP-relative addressing (used 
only in 64-bit mode). Such addressing causes an instruction to use an address that is the sum of the 
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displacement field and the value of RIP that references the following instruction. In this case, the exit 
qualification is loaded with the sum of the displacement field and the appropriate RIP value. 


In all cases, bits of this field beyond the instruction’s address size are undefined. For example, suppose 
that the address-size field in the VM-exit instruction-information field (see Section 23.9.4 and Section 
26.2.5) reports an n-bit address size. Then bits 63:n (bits 31:n on processors that do not support Intel 64 
architecture) of the instruction displacement are undefined. 


For a control-register access, the exit qualification contains information about the access and has the 
format given in Table 26-3. 


For MOV DR, the exit qualification contains information about the instruction and has the format given in 
Table 26-4. 


For an J/O instruction, the exit qualification contains information about the instruction and has the format 
given in Table 26-5. 


For MWAIT, the exit qualification contains a value that indicates whether address-range monitoring 
hardware was armed. The exit qualification is set either to 0 (if address-range monitoring hardware is not 
armed) or to 1 (if address-range monitoring hardware is armed). 


WBINVD and WBNOINVD use the same basic exit reason (see Appendix C). For WBINVD, the exit qualifi- 
cation is 0, while for WBNOINVD it is 1. 


For an APIC-access VM exit resulting from a linear access or a guest-physical access to the APIC-access 
page (see Section 28.4), the exit qualification contains information about the access and has the format 
given in Table 26-6.! 


If the access to the APIC-access page occurred during execution of an instruction in enclave mode (and not 
during delivery of an event incident to enclave mode), bits 11:0 of the exit qualification are cleared. 


Sucha VM exit that set bits 15:12 of the exit qualification to O000b (data read during instruction execution) 
or 0001b (data write during instruction execution) set bit 12—which distinguishes data read from data 
write—to that which would have been stored in bit 1—W/R—of the page-fault error code had the access 
caused a page fault instead of an APIC-access VM exit. This implies the following: 


¢ For an APIC-access VM exit caused by the CLFLUSH and CLFLUSHOPT instructions, the access type is 
“data read during instruction execution.” 


¢ For an APIC-access VM exit caused by the ENTER instruction, the access type is “data write during 
instruction execution.” 


Table 26-3. Exit Qualification for Control-Register Accesses 


Bit Positions 


Contents 


3:0 


Number of control register (0 for CLTS and LMSW). Bit 3 is always O on processors that do not support Intel 64 
architecture as they do not support CR8. 


5:4 


Access type: 


0 = MOV to CR 

1 = MOV from CR 
2=CLTS 

3 = LMSW 


LMSW operand type: 
O = register 
1 = memory 


For CLTS and MOV CR, cleared to 0 


1. The exit qualification is undefined if the access was part of the logging of a branch record or a processor-event-based-sampling 
(PEBS) record to the DS save area. It is recommended that software configure the paging structures so that no address in the DS 
save area translates to an address on the APIC-access page. 
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Table 26-3. Exit Qualification for Control-Register Accesses (Contd.) 


Bit Positions Contents 


Not currently defined 


For MOV CR, the general-purpose register: 
0 = RAX 
1 =RCX 


8-15 represent R8-R15, respectively (used only on processors that support Intel 64 architecture) 


For CLTS and LMSW, cleared to 0 


15:12 


Not currently defined 


31:16 


For LMSW, the LMSW source data 
For CLTS and MOV CR, cleared to 0 


63:32 


Not currently defined. These bits exist only on processors that support Intel 64 architecture. 


¢ Foran APIC-access VM exit caused by the MASKMOVQ instruction or the MASKMOVDQU instruction, the 
access type is “data write during instruction execution.” 


¢ For an APIC-access VM exit caused by the MONITOR instruction, the access type is “data read during 
instruction execution.” 


¢ For an APIC-access VM exit caused directly by an access to a linear address in the DS save area (BTS or 
PEBS), the access type is “linear access for monitoring.” 


¢ For an APIC-access VM exit caused by a guest-physical access performed for an access to the DS save 
area (e.g., to access a paging structure to translate a linear address), the access type is “guest-physical 
access for monitoring or trace.” 


¢ Foran APIC-access VM exit caused by trace-address pre-translation (TAPT) when the “Intel PT uses 
guest physical addresses” VM-execution control is 1, the access type is “guest-physical access for 
monitoring or trace.” 


Such a VM exit stores 1 for bit 31 for IDT-vectoring information field (see Section 26.2.4) if and only if it 
sets bits 15:12 of the exit qualification to 0011b (linear access during event delivery) or 1010b (guest- 
physical access during event delivery). 


See Section 28.4.4 for further discussion of these instructions and APIC-access VM exits. 


For APIC-access VM exits resulting from physical accesses to the APIC-access page (see Section 28.4.6), 
the exit qualification is undefined. 


For an EPT violation, the exit qualification contains information about the access causing the EPT violation 
and has the format given in Table 26-7. 


As noted in that table, the format and meaning of the exit qualification depends on the setting of the 
“mode-based execute control for EPT” VM-execution control and whether the processor supports advanced 
VM-exit information for EPT violations.? 


1. Software can determine whether advanced VM-exit information for EPT violations is supported by consulting the VMX capability 
MSR IA32_VMX_EPT_VPID_CAP (see Appendix A.10). 
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An EPT violation that occurs during as a result of execution of a read-modify-write operation sets bit 1 (data 
write). Whether it also sets bit 0 (data read) is implementation-specific and, for a given implementation, 


may differ for different kinds of read-modify-write operations. 


Table 26-4. Exit Qualification for MOV DR 


Bit Position(s) | Contents 
2:0 Number of debug register 
3 Not currently defined 
4 Direction of access (O = MOV to DR; 1 = MOV from DR) 
7:5 Not currently defined 
11:8 General-purpose register: 
0 = RAX 
1 =RCX 
2 = RDX 
3 = RBX 
4 = RSP 
5 =RBP 
6 = RSI 
7=RDI 
8-15 = R8 - R15, respectively 
63:12 Not currently defined. Bits 63:32 exist only on processors that support Intel 64 architecture. 
Table 26-5. Exit Qualification for I/O Instructions 
Bit Position(s) Contents 
2:0 Size of access: 
0 = 1-byte 
1 = 2-byte 
3 = 4-byte 
Other values not used 
3 Direction of the attempted access (O = OUT, 1 = IN) 
4 String instruction (0 = not string; 1 = string) 
5 REP prefixed (O = not REP; 1 = REP) 
6 Operand encoding (0 = DX, 1 = immediate) 
15:7 Not currently defined 
31:16 Port number (as specified in DX or in an immediate operand) 
63:32 Not currently defined. These bits exist only on processors that support Intel 64 architecture. 


Bit 12 reports “NMI unblocking due to IRET”; see Section 26.2.3. 
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Table 26-6. Exit Qualification for APIC-Access VM Exits from Linear Accesses and Guest-Physical Accesses 


Bit Position(s) 


Contents 


11:0 « If the APIC-access VM exit is due to a linear access, the offset of access within the APIC page. 
« Undefined if the APIC-access VM exit is due a guest-physical access 
15:12 Access type: 
O = linear access for a data read during instruction execution 
1 = linear access for a data write during instruction execution 
2 = linear access for an instruction fetch 
3 = linear access (read or write) during event delivery 
4 = linear access for monitoring 
10 = guest-physical access during event delivery 
11 = guest-physical access for monitoring or trace 
15 = quest-physical access for an instruction fetch or during instruction execution 
Other values not used 
16 This bit is set for certain accesses that are asynchronous to instruction execution and not part of event delivery. 
These includes guest-physical accesses related to trace output by Intel PT (see Section 24.5.4) and accesses 
related to PEBS on processors with the “EPT-friendly” enhancement (see Section 18.9.5). 
63:17 Not currently defined. Bits 63:32 exist only on processors that support Intel 64 architecture. 


Bit 16 is set for certain accesses that are asynchronous to instruction execution and not part of event 
delivery. These include trace-address pre-translation (TAPT) for Intel PT (see Section 24.5.4) and accesses 
related to PEBS on processors with the “EPT-friendly” enhancement (see Section 18.9.5). 


— For VM exits caused as part of EOI virtualization (Section 28.1.4), bits 7:0 of the exit qualification are set 
to vector of the virtual interrupt that was dismissed by the EOI virtualization. Bits above bit 7 are cleared. 


— For APIC-write VM exits (Section 28.4.3.3), bits 11:0 of the exit qualification are set to the page offset of 
the write access that caused the VM exit.! Bits above bit 11 are cleared. 


— Fora VM exit due to a page-modification log-full event (Section 27.2.6), bit 12 of the exit qualification 
reports “NMI unblocking due to IRET.” Bit 16 is set if the VM exit occurs during TAPT or EPT-friendly PEBS. 
All other bits of the exit qualification are undefined. 


— Fora VM exit due to an SPP-related event (Section 27.2.4), bit 11 of the exit qualification indicates the type 
of event: 0 indicates an SPP misconfiguration and 1 indicates an SPP miss. Bit 12 of the exit qualification 
reports “NMI unblocking due to IRET.” Bit 16 is set if the VM exit occurs during TAPT or EPT-friendly PEBS. 
All other bits of the exit qualification are undefined. 


Guest linear address. For some VM exits, this field receives a linear address that pertains to the VM exit. The 
field is set for different VM exits as follows: 


— VM exits due to attempts to execute LMSW with a memory operand. In these cases, this field receives the 
linear address of that operand. Bits 63:32 are cleared if the logical processor was not in 64-bit mode before 
the VM exit. 


— VM exits due to attempts to execute INS or OUTS for which the relevant segment is usable (if the relevant 
segment is not usable, the value is undefined). (ES is always the relevant segment for INS; for OUTS, the 
relevant segment is DS unless overridden by an instruction prefix.) The linear address is the base address 
of relevant segment plus (E)DI (for INS) or (E)SI (for OUTS). Bits 63:32 are cleared if the logical processor 
was not in 64-bit mode before the VM exit. 


— VM exits due to EPT violations that set bit 7 of the exit qualification (see Table 26-7; these are all EPT 
violations except those resulting from an attempt to load the PDPTEs as of execution of the MOV CR 
instruction and those due to TAPT). The linear address may translate to the guest-physical address whose 
access caused the EPT violation. Alternatively, translation of the linear address may reference a paging- 


Execution of WRMSR with ECX = 83FH (self-IP] MSR) can lead to an APIC-write VM exit; the exit qualification for such an APIC-write 


VM exit is 3FOH. 
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Table 26-7. Exit Qualification for EPT Violations 


Bit Position(s) Contents 

0 Set if the access causing the EPT violation was a data read. 

1 Set if the access causing the EPT violation was a data write. | 

2 Set if the access causing the EPT violation was an instruction fetch. 

3 The logical-AND of bit O in the EPT paging-structure entries used to translate the quest-physical address of the 
access Causing the EPT violation (indicates whether the guest-physical address was readable).2 

4 The logical-AND of bit 1 in the EPT paging-structure entries used to translate the quest-physical address of the 
access Causing the EPT violation (indicates whether the guest-physical address was writeable). 

5 The logical-AND of bit 2 in the EPT paging-structure entries used to translate the quest-physical address of the 
access causing the EPT violation. 

If the “mode-based execute control for EPT” VM-execution control is 0, this indicates whether the guest-physical 
address was executable. If that control is 1, this indicates whether the guest-physical address was executable 
for supervisor-mode linear addresses. 

6 If the “mode-based execute control” VM-execution control is 0, the value of this bit is undefined. If that control is 
1, this bit is the logical-AND of bit 10 in the EPT paging-structure entries used to translate the quest-physical 
address of the access causing the EPT violation. In this case, it indicates whether the quest-physical address was 
executable for user-mode linear addresses. 

7 Set if the guest linear-address field is valid. 

The guest linear-address field is valid for all EPT violations except those resulting from an attempt to load the 
guest PDPTEs as part of the execution of the MOV CR instruction and those due to trace-address pre-translation 
(TAPT; Section 24.5.4). 

8 If bit 7 is 1: 

« Set if the access causing the EPT violation is to a quest-physical address that is the translation of a linear 
address. 

* Clear if the access causing the EPT violation is to a paging-structure entry as part of a page walk or the 
update of an accessed or dirty bit. 

Reserved if bit 7 is O (cleared to O). 

9 If bit 7 is 1, bit 8 is 1, and the processor supports advanced VM-exit information for EPT violations,? this bit is O 
if the linear address is a supervisor-mode linear address and 1 if it is a user-mode linear address. (If CRO.PG = 0, 
the translation of every linear address is a user-mode linear address and thus this bit will be 1.) Otherwise, this 
bit is undefined. 

10 If bit 7 is 1, bit 8 is 1, and the processor supports advanced VM-exit information for EPT violations,? this bit is 0 
if paging translates the linear address to a read-only page and 1 if it translates to a read/write page. (If CRO.PG = 
O, every linear address is read/write and thus this bit will be 1.) Otherwise, this bit is undefined. 

11 If bit 7 is 1, bit 8 is 1, and the processor supports advanced VM-exit information for EPT violations,? this bit is O 
if paging translates the linear address to an executable page and 1 if it translates to an execute-disable page. (If 
CRO.PG = 0, CR4.PAE = 0, or IA32_EFER.NXE = 0, every linear address is executable and thus this bit will be 0.) 
Otherwise, this bit is undefined. 

12 NMI unblocking due to IRET (see Section 26.2.3). 

13 Set if the access causing the EPT violation was a shadow-stack access. 

14 If supervisor shadow-stack control is enabled (by setting bit 7 of EPTP), this bit is the same as bit 60 in the EPT 
paging-structure entry that maps the page of the guest-physical address of the access causing the EPT violation. 
Otherwise (or if translation of the guest-physical address terminates before reaching an EPT paging-structure 
entry that maps a page), this bit is undefined. 
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Table 26-7. Exit Qualification for EPT Violations (Contd.) 


Bit Position(s) Contents 


15 Not currently defined. 


16 This bit is set if the access was asynchronous to instruction execution not the result of event delivery. (The bit is 
set if the access is related to trace output by Intel PT; see Section 24.5.4.) Otherwise, this bit is cleared. 


63:17 Not currently defined. Bits 63:32 exist only on processors that support Intel 64 architecture. 


NOTES: 

1. If accessed and dirty flags for EPT are enabled, processor accesses to quest paging-structure entries are treated as writes with 
regard to EPT violations (see Section 27.2.3.2). If such an access causes an EPT violation, the processor sets both bit 0 and bit 1 of 
the exit qualification. 

2. Bits 5:3 are cleared to O if any of EPT paging-structure entries used to translate the quest-physical address of the access causing the 
EPT violation is not present (see Section 27.2.2). 

3. Software can determine whether advanced VM-exit information for EPT violations is supported by consulting the VMX capability 
MSR IA32_VMX_EPT_VPID_CAP (see Appendix A.10). 


structure entry whose access caused the EPT violation. Bits 63:32 are cleared if the logical processor was 
not in 64-bit mode before the VM exit. 


If the EPT violation occurred during execution of an instruction in enclave mode (and not during delivery of 
an event incident to enclave mode), bits 11:0 of this field are cleared. 


— VM exits due to SPP-related events. 
— For all other VM exits, the field is undefined. 


¢ Guest-physical address. For a VM exit due to an EPT violation, an EPT misconfiguration, or an SPP-related 
event, this field receives the guest-physical address that caused the EPT violation or EPT misconfiguration. For 
all other VM exits, the field is undefined. 


If the EPT violation or EPT misconfiguration occurred during execution of an instruction in enclave mode (and 
not during delivery of an event incident to enclave mode), bits 11:0 of this field are cleared. 


26.2.2 Information for VM Exits Due to Vectored Events 


Section 23.9.2 defines fields containing information for VM exits due to the following events: exceptions (including 
those generated by the instructions INT1, INT3, INTO, BOUND, UDO, UD1, and UD2); external interrupts that occur 
while the “acknowledge interrupt on exit” VM-exit control is 1; and non-maskable interrupts (NMIs).1 Such 

VM exits include those that occur on an attempt at a task switch that causes an exception before generating the 
VM exit due to the task switch that causes the VM exit. 


The following items detail the use of these fields: 


¢ VM-exit interruption information (format given in Table 23-17). The following items detail how this field is 
established for VM exits due to these events: 


— Foran exception, bits 7:0 receive the exception vector (at most 31). For an NMI, bits 7:0 are set to 2. For 
an external interrupt, bits 7:0 receive the vector. 


— Bits 10:8 are set to O (external interrupt), 2 (non-maskable interrupt), 3 (hardware exception), 5 
(privileged software exception), or 6 (software exception). Hardware exceptions comprise all exceptions 
except the following: 


* Debug exceptions (#DB) generated by the INT1 instruction; these are privileged software exceptions. 
(Other debug exceptions are considered hardware exceptions, as are those caused by executions of 
INT1 in enclave mode.) 


1. INT1 and INT3 refer to the instructions with opcodes F1 and CC, respectively, and not to INT n with value 1 or 3 for n. 
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¢ Breakpoint exceptions (#BP; generated by INT3) and overflow exceptions (#OF; generated by INTO); 
these are software exceptions. (A #BP that occurs in enclave mode is considered a hardware 
exception.) 


BOUND-range exceeded exceptions (#BR; generated by BOUND) and invalid opcode exceptions (#UD) 
generated by UDO, UD1, and UD2 are hardware exceptions. 


— Bit 11 is set to 1 if the VM exit is caused by a hardware exception that would have delivered an error code 
on the stack. This bit is always 0 if the VM exit occurred while the logical processor was in real-address 
mode (CRO.PE=0).! If bit 11 is set to 1, the error code is placed in the VM-exit interruption error code (see 
below). 


— Bit 12 reports “NMI unblocking due to IRET”; see Section 26.2.3. The value of this bit is undefined if the 
VM exit is due to a double fault (the interruption type is hardware exception and the vector is 8). 


— Bits 30:13 are always set to 0. 
— Bit 31 is always set to 1. 


For other VM exits (including those due to external interrupts when the “acknowledge interrupt on exit” VM-exit 
control is 0), the field is marked invalid (by clearing bit 31) and the remainder of the field is undefined. 


¢ \VM-exit interruption error code. 


— For VM exits that set both bit 31 (valid) and bit 11 (error code valid) in the VM-exit interruption-information 
field, this field receives the error code that would have been pushed on the stack had the event causing the 
VM exit been delivered normally through the IDT. The EXT bit is set in this field exactly when it would be set 
normally. For exceptions that occur during the delivery of double fault (if the IDT-vectoring information field 
indicates a double fault), the EXT bit is set to 1, assuming that (1) that the exception would produce an 
error code normally (if not incident to double-fault delivery) and (2) that the error code uses the EXT bit 
(not for page faults, which use a different format). 


— For other VM exits, the value of this field is undefined. 


26.2.3. Information About NMI Unblocking Due to IRET 


AVM exit may occur during execution of the IRET instruction for reasons including the following: faults, EPT viola- 
tions, page-modification log-full events, or SPP-related events. 


An execution of IRET that commences while non-maskable interrupts (NMIs) are blocked will unblock NMIs even if 
a fault or VM exit occurs; the state saved by such a VM exit will indicate that NMIs were not blocked. 


VM exits for the reasons enumerated above provide more information to software by saving a bit called “NMI 
unblocking due to IRET.” This bit is defined if (1) either the “NMI exiting” VM-execution control is 0 or the “virtual 
NMIs” VM-execution control is 1; (2) the VM exit does not set the valid bit in the IDT-vectoring information field 
(see Section 26.2.4); and (3) the VM exit is not due to a double fault. In these cases, the bit is defined as follows: 


¢ The bit is 1 if the VM exit resulted from a memory access as part of execution of the IRET instruction and one 
of the following holds: 


— The “virtual NMIs” VM-execution control is 0 and blocking by NMI (see Table 23-3) was in effect before 
execution of IRET. 


— The “virtual NMIs” VM-execution control is 1 and virtual-NMI blocking was in effect before execution of 
IRET. 


¢ The bit is 0 for all other relevant VM exits. 


For VM exits due to faults, NMI unblocking due to IRET is saved in bit 12 of the VM-exit interruption-information 
field (Section 26.2.2). For VM exits due to EPT violations, page-modification log-full events, and SPP-related 
events, NMI unblocking due to IRET is saved in bit 12 of the exit qualification (Section 26.2.1). 


1. If the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PE must be 1 in VMX operation, a logical processor cannot be in real- 
address mode unless the “unrestricted guest” VM-execution control and bit 31 of the primary processor-based V/M-execution con- 
trols are both 1. 
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(Executions of IRET may also incur VM exits due to APIC accesses and EPT misconfigurations. These VM exits do 
not report information about NMI unblocking due to IRET.) 


26.2.4 Information for VM Exits During Event Delivery 


Section 23.9.3 defined fields containing information for VM exits that occur while delivering an event through the 
IDT and as a result of any of the following cases: 


A fault occurs during event delivery and causes a VM exit (because the bit associated with the fault is set to 1 
in the exception bitmap). 


A task switch is invoked through a task gate in the IDT. The VM exit occurs due to the task switch only after the 
initial checks of the task switch pass (see Section 24.4.2). 


Event delivery causes an APIC-access VM exit (see Section 28.4). 


An EPT violation, EPT misconfiguration, page-modification log-full event, or SPP-related event that occurs 
during event delivery. 


These fields are used for VM exits that occur during delivery of events injected as part of VM entry (see Section 
25.6.1.2). 


A VM exit is not considered to occur during event delivery in any of the following circumstances: 


The original event causes the VM exit directly (for example, because the original event is a non-maskable 
interrupt (NMI) and the “NMI exiting” VM-execution control is 1). 


The original event results in a double-fault exception that causes the VM exit directly. 
The VM exit occurred as a result of fetching the first instruction of the handler invoked by the event delivery. 
The VM exit is caused by a triple fault. 


The following items detail the use of these fields: 


IDT-vectoring information (format given in Table 23-18). The following items detail how this field is established 
for VM exits that occur during event delivery: 


— Ifthe VM exit occurred during delivery of an exception, bits 7:0 receive the exception vector (at most 31). 
If the VM exit occurred during delivery of an NMI, bits 7:0 are set to 2. If the VM exit occurred during 
delivery of an external interrupt, bits 7:0 receive the vector. 


— Bits 10:8 are set to indicate the type of event that was being delivered when the VM exit occurred: 0 
(external interrupt), 2 (non-maskable interrupt), 3 (hardware exception), 4 (software interrupt), 5 
(privileged software interrupt), or 6 (software exception). 


Hardware exceptions comprise all exceptions except the following: 


* Debug exceptions (#DB) generated by the INT1 instruction; these are privileged software exceptions. 
(Other debug exceptions are considered hardware exceptions, as are those caused by executions of 
INT1 in enclave mode.) 


¢ Breakpoint exceptions (#BP; generated by INT3) and overflow exceptions (#OF; generated by INTO); 
these are software exceptions. (A #BP that occurs in enclave mode is considered a hardware 
exception. ) 


BOUND-range exceeded exceptions (#BR; generated by BOUND) and invalid opcode exceptions (#UD) 
generated by UDO, UD1, and UD2 are hardware exceptions. 


— Bit 11 is set to 1 if the VM exit occurred during delivery of a hardware exception that would have delivered 
an error code on the stack. This bit is always 0 if the VM exit occurred while the logical processor was in 
real-address mode (CRO.PE=0).? If bit 11 is set to 1, the error code is placed in the IDT-vectoring error 
code (see below). 


. This includes the case in which a VM exit occurs while delivering a software interrupt (INT n) through the 16-bit IVT (interrupt vec- 


tor table) that is used in virtual-8086 mode with virtual-machine extensions (if RFLAGS.VM = CR4.VME = 1). 


In the following items, INT1 and INT3 refer to the instructions with opcodes F1 and CC, respectively, and not to INT n with value 1 or 
3 for n. 
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— Bit 12 is undefined. 

— Bits 30:13 are always set to 0. 

— Bit 31 is always set to 1. 

For other VM exits, the field is marked invalid (by clearing bit 31) and the remainder of the field is undefined. 
¢  IDT-vectoring error code. 


— For VM exits that set both bit 31 (valid) and bit 11 (error code valid) in the IDT-vectoring information field, 
this field receives the error code that would have been pushed on the stack by the event that was being 
delivered through the IDT at the time of the VM exit. The EXT bit is set in this field when it would be set 
normally. 


— For other VM exits, the value of this field is undefined. 


26.2.5 ‘Information for VM Exits Due to Instruction Execution 


Section 23.9.4 defined fields containing information for VM exits that occur due to instruction execution. (The VM- 
exit instruction length is also used for VM exits that occur during the delivery of a software interrupt or software 
exception.) The following items detail their use. 


° \VM-exit instruction length. This field is used in the following cases: 


— For fault-like VM exits due to attempts to execute one of the following instructions that cause VM exits 
unconditionally (see Section 24.1.2) or based on the settings of VM-execution controls (see Section 
24.1.3): CLTS, CPUID, ENCLS, GETSEC, HLT, IN, INS, INVD, INVEPT, INVLPG, INVPCID, INVVPID, LGDT, 
LIDT, LLDT, LMSW, LOADIWKEY, LTR, MONITOR, MOV CR, MOV DR, MWAIT, OUT, OUTS, PAUSE, RDMSR, 
RDPMC, RDRAND, RDSEED, RDTSC, RDTSCP, RSM, SGDT, SIDT, SLDT, STR, TPAUSE, UMWAIT, VMCALL, 
VMCLEAR, VMLAUNCH, VMPTRLD, VMPTRST, VMREAD, VMRESUME, VMWRITE, VMXOFF, VMXON, WBINVD, 
WBNOINVD, WRMSR, XRSTORS, XSETBV, and XSAVES.! 


— For VM exits due to software exceptions (those generated by executions of INT3 or INTO) or privileged 
software exceptions (those generated by executions of INT1). 


— For VM exits due to faults encountered during delivery of a software interrupt, privileged software 
exception, or software exception. 


— For VM exits due to attempts to effect a task switch via instruction execution. These are VM exits that 
produce an exit reason indicating task switch and either of the following: 


¢ An exit qualification indicating execution of CALL, IRET, or JMP instruction. 


¢ An exit qualification indicating a task gate in the IDT and an IDT-vectoring information field indicating 
that the task gate was encountered during delivery of a software interrupt, privileged software 
exception, or software exception. 


— For APIC-access VM exits and for VM exits caused by EPT violations, page-modification log-full events, and 
SPP-related events encountered during delivery of a software interrupt, privileged software exception, or 
software exception.2 


— For VM exits due executions of VMFUNC that fail because one of the following is true: 


¢ EAX indicates a VM function that is not enabled (the bit at position EAX is 0 in the VM-function controls; 
see Section 24.5.6.2). 


3. If the capability MSR IA32_VMX_CRO_FIXEDO reports that CRO.PE must be 1 in VMX operation, a logical processor cannot be in real- 
address mode unless the “unrestricted guest” VM-execution control and bit 31 of the primary processor-based V/M-execution con- 
trols are both 1. 

1. This item applies only to fault-like VM exits. It does not apply to trap-like VM exits following executions of the MOV to CR8 instruc- 
tion when the “use TPR shadow” VM-execution control is 1 or to those following executions of the WRMSR instruction when the 
“virtualize x2APIC mode” VM-execution control is 1. 

2. The VM-exit instruction-length field is not defined following APIC-access VM exits resulting from physical accesses (see Section 
28.4.6) even if encountered during delivery of a software interrupt, privileged software exception, or software exception. 
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¢ EAX = 0 and either ECX > 512 or the value of ECX selects an invalid tentative EPTP value (see Section 
24.5.6.3). 


In all the above cases, this field receives the length in bytes (1-15) of the instruction (including any instruction 
prefixes) whose execution led to the VM exit (see the next paragraph for one exception). 


The cases of VM exits encountered during delivery of a software interrupt, privileged software exception, or 
software exception include those encountered during delivery of events injected as part of VM entry (see 
Section 25.6.1.2). If the original event was injected as part of VM entry, this field receives the value of the VM- 
entry instruction length. 


All VM exits other than those listed in the above items leave this field undefined. 
If the VM exit occurred in enclave mode, this field is cleared (none of the previous items apply). 


Table 26-8. Format of the VM-Exit Instruction-Information Field as Used for INS and OUTS 
Bit Position(s) | Content 


6:0 Undefined. 

9:7 Address size: 
0: 16-bit 
1: 32-bit 


2: 64-bit (used only on processors that support Intel 64 architecture) 
Other values not used. 
14:10 Undefined. 
17:15 Segment register: 
0: ES 


Other values not used. Undefined for VM exits due to execution of INS. 
31:18 Undefined. 


° VM-exit instruction information. For VM exits due to attempts to execute INS, INVEPT, INVPCID, INVVPID, 
LIDT, LGDT, LLDT, LOADIWKEY, LTR, OUTS, RDRAND, RDSEED, SIDT, SGDT, SLDT, STR, VMCLEAR, VMPTRLD, 
VMPTRST, VMREAD, VMWRITE, VMXON, XRSTORS, or XSAVES, this field receives information about the 
instruction that caused the VM exit. The format of the field depends on the identity of the instruction causing 
the VM exit: 


— For VM exits due to attempts to execute INS or OUTS, the field has the format is given in Table 26-8. 


— For VM exits due to attempts to execute INVEPT, INVPCID, or INVVPID, the field has the format is given in 
Table 26-9. 


— For VM exits due to attempts to execute LIDT, LGDT, SIDT, or SGDT, the field has the format is given in 
Table 26-10. 


— For VM exits due to attempts to execute LLDT, LTR, SLDT, or STR, the field has the format is given in 
Table 26-11. 


— For VM exits due to attempts to execute RDRAND, RDSEED, TPAUSE, or UMWAIT, the field has the format 
is given in Table 26-12. 


— For VM exits due to attempts to execute VMCLEAR, VMPTRLD, VMPTRST, VMXON, XRSTORS, or XSAVES, 
the field has the format is given in Table 26-13. 


1. The format of the field was undefined for these VM exits on the first processors to support the virtual-machine extensions. Soft- 
ware can determine whether the format specified in Table 26-8 is used by consulting the VMX capability MSR [A32_VMX_BASIC 
(see Appendix A.1). 
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— For VM exits due to attempts to execute VMREAD or VMWRITE, the field has the format is given in 
Table 26-14. 


— For VM exits due to attempts to execute LOADIWKEY, the field has the format is given in Table 26-15. 


For all other VM exits, the field is undefined, unless the VM exit occurred in enclave mode, in which case the 
field is cleared. 


¢ 1/0 RCX, I/O RSI, I/O RDI, I/O RIP. These fields are undefined except for SMM VM exits due to system- 
management interrupts (SMIs) that arrive immediately after retirement of I/O instructions. See Section 


30.15.2.3. 


Note that, if the VM exit occurred in enclave mode, these fields are all cleared. 


Table 26-9. Format of the VM-Exit Instruction-Information Field as Used for INVEPT, INVPCID, and INVVPID 


Bit Position(s) 


Content 


1:0 


Scaling: 
0: no scaling 
1: scale by 2 
2: scale by 4 
3: scale by 8 (used only on processors that support Intel 64 architecture) 


Undefined for instructions with no index register (bit 22 is set). 


6:2 


Undefined. 


9:7 


Address size: 

0: 16-bit 

1: 32-bit 

2: 64-bit (used only on processors that support Intel 64 architecture) 
Other values not used. 


10 


Cleared to 0. 


14:11 


Undefined. 


17:15 


Segment register: 
0: ES 
1:CS 
2:SS 
3: DS 
4: FS 
5:GS 
Other values not used. 


21:18 


IndexReg: 

0 = RAX 

1 =RCX 

2 = RDX 

3 = RBX 

4 =RSP 

5 =RBP 

6 = RSI 

7=RDI 

8-15 represent R8-R15, respectively (used only on processors that support Intel 64 architecture) 
Undefined for instructions with no index register (bit 22 is set). 


22 


IndexReg invalid (O = valid; 1 = invalid) 


26:23 


BaseReg (encoded as IndexReg above) 
Undefined for memory instructions with no base register (bit 27 is set). 


27 
31:28 


BaseReg invalid (0 = valid; 1 = invalid) 
Reg2 (same encoding as IndexReg above) 
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Table 26-10. Format of the VM-Exit Instruction-Information Field as Used for LIDT, LGDT, SIDT, or SGDT 


Bit Position(s) | Content 

1:0 Scaling: 
0: no scaling 
1: scale by 2 
2: scale by 4 


3: scale by 8 (used only on processors that support Intel 64 architecture) 
Undefined for instructions with no index register (bit 22 is set). 


6:2 


Undefined. 


9:7 


Address size: 

0: 16-bit 

1: 32-bit 

2: 64-bit (used only on processors that support Intel 64 architecture) 
Other values not used. 


10 


Cleared to 0. 


11 


Operand size: 
0: 16-bit 
1: 32-bit 
Undefined for VM exits from 64-bit mode. 


14:12 


Undefined. 


17:15 


Segment register: 
0: ES 
1:CS 
2: SS 
3:DS 
4: FS 
5:GS 
Other values not used. 


21:18 


IndexReg: 

0 = RAX 

1 =RCX 

2 = RDX 

3 = RBX 

4 =RSP 

5 = RBP 

6 =RSI 

7=RDI 

8-15 represent R8-R15, respectively (Used only on processors that support Intel 64 architecture) 
Undefined for instructions with no index register (bit 22 is set). 


22 


IndexReg invalid (0 = valid; 1 = invalid) 


26:23 


BaseReg (encoded as IndexReg above) 
Undefined for instructions with no base register (bit 27 is set). 


27 


BaseReg invalid (0 = valid; 1 = invalid) 


29:28 


Instruction identity: 


0: SGDT 
1: SIDT 
2: LGDT 
3: LIDT 
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Table 26-10. 


Format of the VM-Exit Instruction-Information Field as Used for LIDT, LGDT, SIDT, or SGDT (Contd.) 


Bit Position(s) 


Content 


31:30 


Undefined. 


Table 26-11. Format of the VM-Exit Instruction-Information Field as Used for LLDT, LTR, SLDT, and STR 


Bit Position(s) 


Content 


1:0 


Scaling: 
0: no scaling 
1: scale by 2 
2: scale by 4 
3: scale by 8 (used only on processors that support Intel 64 architecture) 


Undefined for register instructions (bit 10 is set) and for memory instructions with no index register (bit 10 is clear 
and bit 22 is set). 


Undefined. 
Reg1: 

0 = RAX 

1 =RCX 

2 = RDX 

3 = RBX 

4 = RSP 

5 = RBP 

6 = RSI 

7 =RDI 

8-15 represent R8-R15, respectively (used only on processors that support Intel 64 architecture) 
Undefined for memory instructions (bit 10 is clear). 


9:7 


Address size: 

0: 16-bit 

1: 32-bit 

2: 64-bit (used only on processors that support Intel 64 architecture) 
Other values not used. Undefined for register instructions (bit 10 is set). 


10 


Mem/Reg (0 = memory; 1 = register). 


14:11 


Undefined. 


17:15 


21:18 


Segment register: 

0: ES 

1:¢S 

2:SS 

3: DS 

4: FS 

5: GS 
Other values not used. Undefined for register instructions (bit 10 is set). 
IndexReg (encoded as Reg1 above) 


Undefined for register instructions (bit 10 is set) and for memory instructions with no index register (bit 10 is clear 
and bit 22 is set). 


22 


IndexReg invalid (0 = valid; 1 = invalid) 
Undefined for register instructions (bit 10 is set). 


26:23 


26-18 Vol. 3C 


BaseReg (encoded as Reg1 above) 


Undefined for register instructions (bit 10 is set) and for memory instructions with no base register (bit 10 is clear 
and bit 27 is set). 
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Format of the VM-Exit Instruction-Information Field as Used for LLDT, LTR, SLDT, and STR (Contd.) 


Bit Position(s) 


Content 


27 


BaseReg invalid (0 = valid; 1 = invalid) 
Undefined for register instructions (bit 10 is set). 


29:28 


Instruction identity: 


0: SLDT 
1:STR 
2: LLDT 
3: LTR 


31:30 


Undefined. 


Table 26-12. Format of the VM-Exit Instruction-Information Field as Used for RDRAND, RDSEED, TPAUSE, and 


UMWAIT 


Bit Position(s) 


Content 


2:0 


Undefined. 


6:3 


Operand register (destination for RDRAND and RDSEED; source for TPAUSE and UMWAIT): 


8-15 represent R8-R15, respectively (used only on processors that support Intel 64 architecture) 


10:7 


Undefined. 


12:11 


Operand size: 
0: 16-bit 
1: 32-bit 
2: 64-bit 
The value 3 is not used. 


31:13 


Undefined. 


Table 26-13. Format of the VM-Exit Instruction-Information Field as Used for VMCLEAR, VMPTRLD, VMPTRST, 


VMXON, XRSTORS, and XSAVES 


Bit Position(s) 


Content 


1:0 


Scaling: 
0: no scaling 
1: scale by 2 
2: scale by 4 
3: scale by 8 (used only on processors that support Intel 64 architecture) 


Undefined for instructions with no index register (bit 22 is set). 


6:2 


Undefined. 
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Table 26-13. Format of the VM-Exit Instruction-Information Field as Used for VMCLEAR, VMPTRLD, VMPTRST, 


VMXON, XRSTORS, and XSAVES (Contd.) 


Bit Position(s) 


Content 


9:7 


Address size: 

0: 16-bit 

1: 32-bit 

2: 64-bit (used only on processors that support Intel 64 architecture) 
Other values not used. 


10 


Cleared to 0. 


14:11 


Undefined. 


17:15 


Segment register: 
0: ES 
1:CS 
2:SS 
3: DS 
4: FS 
5:GS 
Other values not used. 


21:18 


IndexReg: 


7 =RDI 
8-15 represent R8-R15, respectively (Used only on processors that support Intel 64 architecture) 


Undefined for instructions with no index register (bit 22 is set). 


22 


IndexReg invalid (0 = valid; 1 = invalid) 


26:23 


27 


BaseReg (encoded as IndexReg above) 
Undefined for instructions with no base register (bit 27 is set). 
BaseReg invalid (0 = valid; 1 = invalid) 


31:28 


Undefined. 


Table 26-14. Format of the VM-Exit Instruction-Information Field as Used for VMREAD and VMWRITE 


Bit Position(s) | Content 

1:0 Scaling: 
0: no scaling 
1: scale by 2 
2: scale by 4 


3: scale by 8 (used only on processors that support Intel 64 architecture) 


Undefined for register instructions (bit 10 is set) and for memory instructions with no index register (bit 10 is clear 
and bit 22 is set). 


Undefined. 
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Table 26-14. Format of the VM-Exit Instruction-Information Field as Used for VMREAD and VMWRITE (Contd.) 


Bit Position(s) 


Content 


6:3 


Reg1: 

0 = RAX 

1 =RCX 

2 = RDX 

3 = RBX 

4 =RSP 

5 = RBP 

6 =RSI 

7 =RDI 

8-15 represent R8-R15, respectively (used only on processors that support Intel 64 architecture) 
Undefined for memory instructions (bit 10 is clear). 


9:7 Address size: 
0: 16-bit 
1: 32-bit 
2: 64-bit (used only on processors that support Intel 64 architecture) 
Other values not used. Undefined for register instructions (bit 10 is set). 
10 Mem/Reg (0 = memory; 1 = register). 
14:11 Undefined. 
17:15 Segment register: 
0: ES 
1:¢S 
2:SS 
3: DS 
4: FS 
5: GS 
Other values not used. Undefined for register instructions (bit 10 is set). 
21:18 IndexReg (encoded as Reg1 above) 
Undefined for register instructions (bit 10 is set) and for memory instructions with no index register (bit 10 is clear 
and bit 22 is set). 
22 IndexReg invalid (0 = valid; 1 = invalid) 
Undefined for register instructions (bit 10 is set). 
26:23 BaseReg (encoded as Reg1 above) 
Undefined for register instructions (bit 10 is set) and for memory instructions with no base register (bit 10 is clear 
and bit 27 is set). 
27 BaseReg invalid (0 = valid; 1 = invalid) 
Undefined for register instructions (bit 10 is set). 
31:28 Reg2 (same encoding as Reg1 above) 
Table 26-15. Format of the VM-Exit Instruction-Information Field as Used for LOADIWKEY 
Bit Position(s) | Content 
2:0 Undefined. 
6:3 Reg1: identifies the first XMM register operand (KMMO-XMM15; values 8-15 are used only on processors that 
support Intel 64 architecture). 
30:7 Undefined. 
31:28 Reg2: identifies the second XMM register operand (see above). 
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26.3. SAVING GUEST STATE 


VM exits save certain components of processor state into corresponding fields in the guest-state area of the VMCS 
(see Section 23.4). On processors that support Intel 64 architecture, the full value of each natural-width field (see 
Section 23.11.2) is saved regardless of the mode of the logical processor before and after the VM exit. 


In general, the state saved is that which was in the logical processor at the time the VM exit commences. See 
Section 26.1 for a discussion of which architectural updates occur at that time. 


Section 26.3.1 through Section 26.3.4 provide details for how various components of processor state are saved. 
These sections reference VMCS fields that correspond to processor state. Unless otherwise stated, these references 
are to fields in the guest-state area. 


26.3.1 Saving Control Registers, Debug Registers, and MSRs 
Contents of certain control registers, debug registers, and MSRs is saved as follows: 


¢ The contents of CRO, CR3, CR4, and the IA32_SYSENTER_CS, IA32_SYSENTER_ESP, and IA32_SYSENTER_EIP 
MSRs are saved into the corresponding fields. Bits 63:32 of the IA32_SYSENTER_CS MSR are not saved. On 
processors that do not support Intel 64 architecture, bits 63:32 of the IA32_SYSENTER_ESP and 
IA32_SYSENTER_EIP MSRs are not saved. 


e If the “save debug controls” VM-exit control is 1, the contents of DR7 and the IA32_DEBUGCTL MSR are saved 
into the corresponding fields. The first processors to support the virtual-machine extensions supported only the 
1-setting of this control and thus always saved data into these fields. 


° If the “save IA32_PAT” VM-exit control is 1, the contents of the IA32_PAT MSR are saved into the corresponding 
field. 


e If the “save IA32_EFER” VM-exit control is 1, the contents of the IA32_EFER MSR are saved into the corre- 
sponding field. 


¢ If the processor supports either the 1-setting of the “load IA32_BNDCFGS” VM-entry control or that of the 
“clear IA32_BNDCFGS” VM-exit control, the contents of the IA32_BNDCFGS MSR are saved into the corre- 
sponding field. 


° Ifthe processor supports either the 1-setting of the “load IA32_RTIT_CTL” VM-entry control or that of the “clear 
IA32_RTIT_CTL” VM-exit control, the contents of the IA32_RTIT_CTL MSR are saved into the corresponding 
field. 


e If the processor supports the 1-setting of the “load CET” VM-entry control, the contents of the IA32_S_CET and 
IA32_INTERRUPT_SSP_TABLE_ADDR MSRs are saved into the corresponding fields. On processors that do not 
support Intel 64 architecture, bits 63:32 of these MSRs are not saved. 


° If the processor supports the 1-setting of the “load PKRS” VM-entry control, the contents of the IA32_PKRS 
MSR are saved into the corresponding field. 


¢ The value of the SMBASE field is undefined after all VM exits except SMM VM exits. See Section 30.15.2. 


26.3.2 Saving Segment Registers and Descriptor-Table Registers 


For each segment register (CS, SS, DS, ES, FS, GS, LDTR, or TR), the values saved for the base-address, segment- 
limit, and access rights are based on whether the register was unusable (see Section 23.4.1) before the VM exit: 


e If the register was unusable, the values saved into the following fields are undefined: (1) base address; 
(2) segment limit; and (3) bits 7:0 and bits 15:12 in the access-rights field. The following exceptions apply: 


= CS: 

¢ The base-address and segment-limit fields are saved. 

¢ TheL, D, and G bits are saved in the access-rights field. 
— SS. 

¢ DPL is saved in the access-rights field. 
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* On processors that support Intel 64 architecture, bits 63:32 of the value saved for the base address are 
always zero. 


DS and ES. On processors that support Intel 64 architecture, bits 63:32 of the values saved for the base 
addresses are always zero. 


FS and GS. The base-address field is saved. 
LDTR. The value saved for the base address is always canonical. 


¢ Ifthe register was not unusable, the values saved into the following fields are those which were in the register 
before the VM exit: (1) base address; (2) segment limit; and (3) bits 7:0 and bits 15:12 in access rights. 


¢ Bits 31:17 and 11:8 in the access-rights field are always cleared. Bit 16 is set to 1 if and only if the segment is 
unusable. 


The contents of the GDTR and IDTR registers are saved into the corresponding base-address and limit fields. 


26.3.3 


Saving RIP, RSP, RFLAGS, and SSP 


The contents of the RIP, RSP, RFLAGS, and SSP (shadow-stack pointer) registers are saved as follows: 


° The value saved in the RIP field is determined by the nature and cause of the VM exit: 


If the VM exit occurred in enclave mode, the value saved is the AEP of interrupted enclave thread (the 
remaining items do not apply). 


If the VM exit occurs due to by an attempt to execute an instruction that causes VM exits unconditionally or 
that has been configured to cause a VM exit via the VM-execution controls, the value saved references that 
instruction. 


If the VM exit is caused by an occurrence of an INIT signal, a start-up IPI (SIPI), or system-management 
interrupt (SMI), the value saved is that which was in RIP before the event occurred. 


If the VM exit occurs due to the 1-setting of either the “interrupt-window exiting” VM-execution control or 
the “NMI-window exiting” VM-execution control, the value saved is that which would be in the register had 
the VM exit not occurred. 


If the VM exit is due to an external interrupt, non-maskable interrupt (NMI), or hardware exception (as 
defined in Section 26.2.2), the value saved is the return pointer that would have been saved (either on the 
stack had the event been delivered through a trap or interrupt gate,! or into the old task-state segment had 
the event been delivered through a task gate). 


If the VM exit is due to a triple fault, the value saved is the return pointer that would have been saved 
(either on the stack had the event been delivered through a trap or interrupt gate, or into the old task-state 
segment had the event been delivered through a task gate) had delivery of the double fault not 
encountered the nested exception that caused the triple fault. 


If the VM exit is due to a software exception (due to an execution of INT3 or INTO) or a privileged software 
exception (due to an execution of INT1), the value saved references the INT3, INTO, or INT1 instruction 
that caused that exception. 


Suppose that the VM exit is due to a task switch that was caused by execution of CALL, IRET, or JMP or by 
execution of a software interrupt (INT n), software exception (due to execution of INT3 or INTO), or 
privileged software exception (due to execution of INT1) that encountered a task gate in the IDT. The value 
saved references the instruction that caused the task switch (CALL, IRET, JMP, INT n, INT3, INTO, INT1). 


Suppose that the VM exit is due to a task switch that was caused by a task gate in the IDT that was 
encountered for any reason except the direct access by a software interrupt or software exception. The 
value saved is that which would have been saved in the old task-state segment had the task switch 
completed normally. 


1. The reference here is to the full value of RIP before any truncation that would occur had the stack width been only 32 bits or 16 


bits. 
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— Ifthe VM exit is due to an execution of MOV to CR8 or WRMSR that reduced the value of bits 7:4 of VTPR 
(see Section 28.1.1) below that of TPR threshold VM-execution control field (see Section 28.1.2), the value 
saved references the instruction following the MOV to CR8 or WRMSR. 


— Ifthe VM exit was caused by APIC-write emulation (see Section 28.4.3.2) that results from an APIC access 
as part of instruction execution, the value saved references the instruction following the one whose 
execution caused the APIC-write emulation. 


The contents of the RSP register are saved into the RSP field. 


With the exception of the resume flag (RF; bit 16), the contents of the RFLAGS register is saved into the 
RFLAGS field. RFLAGS.RF is saved as follows: 


— Ifthe VM exit occurred in enclave mode, the value saved is 0 (the remaining items do not apply). 


— Ifthe VM exit is caused directly by an event that would normally be delivered through the IDT, the value 
saved is that which would appear in the saved RFLAGS image (either that which would be saved on the 
stack had the event been delivered through a trap or interrupt gate! or into the old task-state segment had 
the event been delivered through a task gate) had the event been delivered through the IDT. See below for 
VM exits due to task switches caused by task gates in the IDT. 


— Ifthe VM exit is caused by a triple fault, the value saved is that which the logical processor would have in 
RF in the RFLAGS register had the triple fault taken the logical processor to the shutdown state. 


— Ifthe VM exit is caused by a task switch (including one caused by a task gate in the IDT), the value saved 
is that which would have been saved in the RFLAGS image in the old task-state segment (TSS) had the task 
switch completed normally without exception. 


— Ifthe VM exit is caused by an attempt to execute an instruction that unconditionally causes VM exits or one 
that was configured to do with a VM-execution control, the value saved is 0.2 


— For APIC-access VM exits and for VM exits caused by EPT violations, EPT misconfigurations, page-modifi- 
cation log-full events, or SPP-related events, the value saved depends on whether the VM exit occurred 
during delivery of an event through the IDT: 


¢ Ifthe VM exit stored 0 for bit 31 for IDT-vectoring information field (because the VM exit did not occur 
during delivery of an event through the IDT; see Section 26.2.4), the value saved is 1. 


¢ If the VM exit stored 1 for bit 31 for IDT-vectoring information field (because the VM exit did occur 
during delivery of an event through the IDT), the value saved is the value that would have appeared in 
the saved RFLAGS image had the event been delivered through the IDT (see above). 


— For all other VM exits, the value saved is the value RFLAGS.RF had before the VM exit occurred. 


If the processor supports the 1-setting of the “load CET” VM-entry control, the contents of the SSP register are 
saved into the SSP field. 


26.3.4 Saving Non-Register State 


Information corresponding to guest non-register state is saved as follows: 


The activity-state field is saved with the logical processor’s activity state before the VM exit.? See Section 26.1 
for details of how events leading to a VM exit may affect the activity state. 


The interruptibility-state field is saved to reflect the logical processor’s interruptibility before the VM exit. 
— See Section 26.1 for details of how events leading to a VM exit may affect this state. 


. The reference here is to the full value of RFLAGS before any truncation that would occur had the stack width been only 32 bits or 


16 bits. 


This is true even if RFLAGS.RF was 1 before the instruction was executed. If, in response to such a VM exit, a VM monitor re-enters 
the guest to re-execute the instruction that caused the VM exit (for example, after clearing the VM-execution control that caused 
the VM exit), the instruction may encounter a code breakpoint that has already been processed. A VM monitor can avoid this by set- 
ting the quest value of RFLAGS.RF to 1 before resuming guest software. 


If this activity state was an inactive state resulting from execution of a specific instruction (HLT or MWAIT), the value saved for RIP 
by that VM exit will reference the following instruction. 
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VM exits that end outside system-management mode (SMM) save bit 2 (blocking by SMI) as 0 regardless 
of the state of such blocking before the VM exit. 


Bit 3 (blocking by NMI) is treated specially if the “virtual NMIs” VM-execution control is 1. In this case, the 
value saved for this field does not indicate the blocking of NMIs but rather the state of virtual-NMI blocking. 


Bit 4 (enclave interruption) is set to 1 if the VM exit occurred while the logical processor was in enclave 
mode. 


Such VM exits includes those caused by interrupts, non-maskable interrupts, system-management 
interrupts, INIT signals, and exceptions occurring in enclave mode as well as exceptions encountered 
during the delivery of such events incident to enclave mode. 


AVM exit that is incident to delivery of an event injected by VM entry leaves this bit unmodified. 


The pending debug exceptions field is saved as clear for all VM exits except the following: 


A VM exit caused by an INIT signal, a machine-check exception, or a system-management interrupt (SMI). 


A VM exit with basic exit reason “TPR below threshold”,! “virtualized EOI”, “APIC write”, or “monitor trap 
flag.” 


A VM exit due to trace-address pre-translation (TAPT; see Section 24.5.4) or due to accesses related to 
PEBS on processors with the “EPT-friendly” enhancement (see Section 18.9.5). Such VM exits can have 
basic exit reason “APIC access,” “EPT violation,” “EPT misconfiguration,” “page-modification log full,” or 
“SPP-related event.” When due to TAPT or PEBS, these VM exits (with the exception of those due to EPT 
misconfigurations) set bit 16 of the exit qualification, indicating that they are asynchronous to instruction 
execution and not part of event delivery. 


VM exits that are not caused by debug exceptions and that occur while there is MOV-SS blocking of debug 
exceptions. 


For VM exits that do not clear the field, the value saved is determined as follows: 


Each of bits 3:0 may be set if it corresponds to a matched breakpoint. This may be true even if the corre- 
sponding breakpoint is not enabled in DR7. 


Suppose that a VM exit is due to an INIT signal, a machine-check exception, or an SMI; or that a VM exit 
has basic exit reason “TPR below threshold” or “monitor trap flag.” In this case, the value saved sets bits 
corresponding to the causes of any debug exceptions that were pending at the time of the VM exit. 


If the VM exit occurs immediately after VM entry, the value saved may match that which was loaded on 
VM entry (see Section 25.7.3). Otherwise, the following items apply: 


¢ Bit 12 (enabled breakpoint) is set to 1 in any of the following cases: 
— If there was at least one matched data or I/O breakpoint that was enabled in DR7. 


— If it had been set on VM entry, causing there to be valid pending debug exceptions (see Section 
25.7.3) and the VM exit occurred before those exceptions were either delivered or lost. 


— Ifthe XBEGIN instruction was executed immediately before the VM exit and advanced debugging of 
RTM transactional regions had been enabled (see Section 16.3.7, “RTM-Enabled Debugger 
Support,” of Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1). (This does 
not apply to VM exits with basic exit reason “monitor trap flag.”) 


In other cases, bit 12 is cleared to 0. 
¢ Bit 14 (BS) is set if RFLAGS.TF = 1 in either of the following cases: 


— IA32_DEBUGCTL.BTF = 0 and the cause of a pending debug exception was the execution of a single 
instruction. 


— JIA32_DEBUGCTL.BTF = 1 and the cause of a pending debug exception was a taken branch. 


¢ Bit 16 (RTM) is set if a debug exception (#DB) or a breakpoint exception (#BP) occurred inside an RTM 
region while advanced debugging of RTM transactional regions had been enabled. (This does not apply 
to VM exits with basic exit reason “monitor trap flag.”) 


1. This item includes VM exits that occur as a result of certain VM entries (Section 25.7.7). 
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— Suppose that a VM exit is due to another reason (but not a debug exception) and occurs while there is MOV- 
SS blocking of debug exceptions. In this case, the value saved sets bits corresponding to the causes of any 
debug exceptions that were pending at the time of the VM exit. If the VM exit occurs immediately after 
VM entry (no instructions were executed in VMX non-root operation), the value saved may match that 
which was loaded on VM entry (see Section 25.7.3). Otherwise, the following items apply: 


¢ Bit 12 (enabled breakpoint) is set to 1 if there was at least one matched data or I/O breakpoint that was 
enabled in DR7. Bit 12 is also set if it had been set on VM entry, causing there to be valid pending debug 
exceptions (see Section 25.7.3) and the VM exit occurred before those exceptions were either delivered 
or lost. In other cases, bit 12 is cleared to 0. 


¢ The setting of bit 14 (BS) is implementation-specific. However, it is not set if RFLAGS.TF = 0 or 
IA32_DEBUGCTL.BTF = 1. 


— The reserved bits in the field are cleared. 


If the “save VMX-preemption timer value” VM-exit control is 1, the value of timer is saved into the VMX- 
preemption timer-value field. This is the value loaded from this field on VM entry as subsequently decremented 
(see Section 24.5.1). VM exits due to timer expiration save the value 0. Other VM exits may also save the value 
0 if the timer expired during VM exit. (If the “save VMX-preemption timer value” VM-exit control is 0, VM exit 
does not modify the value of the VMX-preemption timer-value field. ) 


If the logical processor supports the 1-setting of the “enable EPT” VM-execution control, values are saved into 
the four (4) PDPTE fields as follows: 


— Ifthe “enable EPT” VM-execution control is 1 and the logical processor was using PAE paging at the time of 
the VM exit, the PDPTE values currently in use are saved:! 


¢ The values saved into bits 11:9 of each of the fields is undefined. 


¢ If the value saved into one of the fields has bit 0 (present) clear, the value saved into bits 63:1 of that 
field is undefined. That value need not correspond to the value that was loaded by VM entry or to any 
value that might have been loaded in VMX non-root operation. 


¢ If the value saved into one of the fields has bit 0 (present) set, the value saved into bits 63:12 of the 
field is a guest-physical address. 


— Ifthe “enable EPT” VM-execution control is 0 or the logical processor was not using PAE paging at the time 
of the VM exit, the values saved are undefined. 


26.4 SAVING MSRS 


After processor state is saved to the guest-state area, values of MSRs may be stored into the VM-exit MSR-store 
area (see Section 23.7.2). Specifically each entry in that area (up to the number specified in the VM-exit MSR-store 
count) is processed in order by storing the value of the MSR indexed by bits 31:0 (as they would be read by 
RDMSR) into bits 127:64. Processing of an entry fails in either of the following cases: 


The value of bits 31:8 is OOO008H, meaning that the indexed MSR is one that allows access to an APIC register 
when the local APIC is in x2APIC mode. 

The value of bits 31:0 indicates an MSR that can be read only in system-management mode (SMM) and the 
VM exit will not end in SMM. (IA32_SMBASE is an MSR that can be read only in SMM.) 

The value of bits 31:0 indicates an MSR that cannot be saved on VM exits for model-specific reasons. A 
processor may prevent certain MSRs (based on the value of bits 31:0) from being stored on VM exits, even if 
they can normally be read by RDMSR. Such model-specific behavior is documented in Chapter 2, “Model- 
Specific Registers (MSRs)” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 


Bits 63:32 of the entry are not all 0. 


A logical processor uses PAE paging if CRO.PG = 1, CR4.PAE = 1 and IA32_EFER.LMA = 0. See Section 4.4 in the Intel® 64 and IA-32 
Architectures Software Developer's Manual, Volume 3A. “Enable EPT” is a secondary processor-based VM-execution control. If bit 31 
of the primary processor-based VM-execution controls is 0, VM exit functions as if the “enable EPT” VM-execution control were O. 
See Section 23.6.2. 
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¢ An attempt to read the MSR indexed by bits 31:0 would cause a general-protection exception if executed via 
RDMSR with CPL = 0. 


A VMX abort occurs if processing fails for any entry. See Section 26.7. 


26.5 LOADING HOST STATE 


Processor state is updated on VM exits in the following ways: 

° Some state is loaded from or otherwise determined by the contents of the host-state area. 
¢ Some state is determined by VM-exit controls. 

¢ Some state is established in the same way on every VM exit. 

° The page-directory pointers are loaded based on the values of certain control registers. 
This loading may be performed in any order. 


On processors that support Intel 64 architecture, the full values of each 64-bit field loaded (for example, the base 
address for GDTR) is loaded regardless of the mode of the logical processor before and after the VM exit. 


The loading of host state is detailed in Section 26.5.1 to Section 26.5.5. These sections reference VMCS fields that 
correspond to processor state. Unless otherwise stated, these references are to fields in the host-state area. 


A logical processor is in IA-32e mode after a VM exit only if the “host address-space size” VM-exit control is 1. If 
the logical processor was in IA-32e mode before the VM exit and this control is 0, a VMX abort occurs. See Section 
26.7. 


In addition to loading host state, VM exits clear address-range monitoring (Section 26.5.6). 


After the state loading described in this section, VM exits may load MSRs from the VM-exit MSR-load area (see 
Section 26.6). This loading occurs only after the state loading described in this section. 


26.5.1 Loading Host Control Registers, Debug Registers, MSRs 


VM exits load new values for controls registers, debug registers, and some MSRs: 


¢ CRO, CR3, and CR4 are loaded from the CRO field, the CR3 field, and the CR4 field, respectively, with the 
following exceptions: 


— The following bits are not modified: 


¢ For CRO, ET, CD, NW; bits 63:32 (on processors that support Intel 64 architecture), 28:19, 17, and 
15:6; and any bits that are fixed in VMX operation (see Section 22.8).! 


¢ For CR3, bits 63:52 and bits in the range 51:32 beyond the processor’s physical-address width (they 
are cleared to 0).2 (This item applies only to processors that support Intel 64 architecture.) 


¢ For CR4, any bits that are fixed in VMX operation (see Section 22.8). 
— CR4.PAE is set to 1 if the “host address-space size” VM-exit control is 1. 
— CR4.PCIDE is set to 0 if the “host address-space size” VM-exit control is 0. 
° DR7 is set to 400H. 
¢ The following MSRs are established as follows: 
— The IA32_DEBUGCTL MSR is cleared to O0O000000_O0000000H. 


— The IA32_SYSENTER_CS MSR is loaded from the IA32_SYSENTER_CS field. Since that field has only 32 
bits, bits 63:32 of the MSR are cleared to 0. 


1. Bits 28:19, 17, and 15:6 of CRO and CRO.ET are unchanged by executions of MOV to CRO. CRO.ET is always 1 and the other bits are 
always 0. 


2. Software can determine a processor's physical-address width by executing CPUID with BOOOOOO8H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 
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The IA32_SYSENTER_ESP and IA32_SYSENTER_EIP MSRs are loaded from the IA32_SYSENTER_ESP and 
IA32_SYSENTER_EIP fields, respectively. 


If the processor does not support the Intel 64 architecture, these fields have only 32 bits; bits 63:32 of the 
MSRs are cleared to 0. 


If the processor supports the Intel 64 architecture with N < 64 linear-address bits, each of bits 63:N is set 
to the value of bit N-1.! 


The following steps are performed on processors that support Intel 64 architecture: 


¢ The MSRs FS.base and GS.base are loaded from the base-address fields for FS and GS, respectively 
(see Section 26.5.2). 


¢ The LMA and LME bits in the IA32_EFER MSR are each loaded with the setting of the “host address- 
space size” VM-exit control. 


If the “load IA32_PERF_GLOBAL_CTRL” VM-exit control is 1, the IA32_PERF_GLOBAL_CTRL MSR is loaded 
from the IA32_PERF_GLOBAL_CTRL field. Bits that are reserved in that MSR are maintained with their 
reserved values. 


If the “load IA32_PAT” VM-exit control is 1, the IA32_PAT MSR is loaded from the IA32_PAT field. Bits that 
are reserved in that MSR are maintained with their reserved values. 


If the “load IA32_EFER” VM-exit control is 1, the IA32_EFER MSR is loaded from the IA32_EFER field. Bits 
that are reserved in that MSR are maintained with their reserved values. 


If the “clear IA32_BNDCFGS” VM-exit control is 1, the IA32_BNDCFGS MSR is cleared to 
00000000_00000000H; otherwise, it is not modified. 


If the “clear IA32_RTIT_CTL” VM-exit control is 1, the IA32_RTIT_CTL MSR is cleared to 
00000000_00000000H; otherwise, it is not modified. 


If the “load CET” VM-exit control is 1, the IA32_S_CET and IA32_INTERRUPT_SSP_TABLE_ADDR MSRs are 
loaded from the IA32_S_CET and IA32_INTERRUPT_SSP_TABLE_ADDR fields, respectively. 


If the processor does not support the Intel 64 architecture, these fields have only 32 bits; bits 63:32 of the 
MSRs are cleared to 0. 


If the processor supports the Intel 64 architecture with N < 64 linear-address bits, each of bits 63:N is set 
to the value of bit N-1. 


If the “load PKRS” VM-exit control is 1, the IA32_PKRS MSR is loaded from the IA32_PKRS field. Bits 63:32 
of that MSR are maintained with zeroes. 


With the exception of FS.base and GS.base, any of these MSRs is subsequently overwritten if it appears in the 
VM-exit MSR-load area. See Section 26.6. 


26.5.2 Loading Host Segment and Descriptor-Table Registers 
Each of the registers CS, SS, DS, ES, FS, GS, and TR is loaded as follows (see below for the treatment of LDTR): 


The selector is loaded from the selector field. The segment is unusable if its selector is loaded with zero. The 
checks specified Section 25.3.1.2 limit the selector values that may be loaded. In particular, CS and TR are 
never loaded with zero and are thus never unusable. SS can be loaded with zero only on processors that 
support Intel 64 architecture and only if the VM exit is to 64-bit mode (64-bit mode allows use of segments 
marked unusable). 


The base address is set as follows: 


CS. Cleared to zero. 
SS, DS, and ES. Undefined if the segment is unusable; otherwise, cleared to zero. 


FS and GS. Undefined (but, on processors that support Intel 64 architecture, canonical) if the segment is 
unusable and the VM exit is not to 64-bit mode; otherwise, loaded from the base-address field. 


Software can determine the number N by executing CPUID with BOOOO008H in EAX. The number of linear-address bits supported is 
returned in bits 15:8 of EAX. 
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If the processor supports the Intel 64 architecture and the processor supports N < 64 linear-address bits, 
each of bits 63:N is set to the value of bit N-1.1 The values loaded for base addresses for FS and GS are 
also manifest in the FS.base and GS.base MSRs. 


— TR. Loaded from the host-state area. If the processor supports the Intel 64 architecture and the processor 
supports N < 64 linear-address bits, each of bits 63:N is set to the value of bit N-1. 


The segment limit is set as follows: 

— CS. Set to FFFFFFFFH (corresponding to a descriptor limit of FFFFFH and a G-bit setting of 1). 
— SS, DS, ES, FS, and GS. Undefined if the segment is unusable; otherwise, set to FFFFFFFFH. 
— TR. Set to 00000067H. 

The type field and S bit are set as follows: 

— CS. Type set to 11 and S set to 1 (execute/read, accessed, non-conforming code segment). 


— SS, DS, ES, FS, and GS. Undefined if the segment is unusable; otherwise, type set to 3 and S set to 1 
(read/write, accessed, expand-up data segment). 


— TR. Type set to 11 and S set to 0 (busy 32-bit task-state segment). 

The DPL is set as follows: 

— CS, SS, and TR. Set to 0. The current privilege level (CPL) will be 0 after the VM exit completes. 
— DS, ES, FS, and GS. Undefined if the segment is unusable; otherwise, set to 0. 

The P bit is set as follows: 

— CS, TR. Set to 1. 

— SS, DS, ES, FS, and GS. Undefined if the segment is unusable; otherwise, set to 1. 


On processors that support Intel 64 architecture, CS.L is loaded with the setting of the “host address-space 
size” VM-exit control. Because the value of this control is also loaded into IA32_EFER.LMA (see Section 26.5.1), 
no VM exit is ever to compatibility mode (which requires IA32_EFER.LMA = 1 and CS.L = 0). 


D/B. 


— CS. Loaded with the inverse of the setting of the “host address-space size” VM-exit control. For example, if 
that control is 0, indicating a 32-bit guest, CS.D/B is set to 1. 


— SS. Set to 1. 
— DS, ES, FS, and GS. Undefined if the segment is unusable; otherwise, set to 1. 
— TR. Set to 0. 


— CS. Setto1. 
— SS, DS, ES, FS, and GS. Undefined if the segment is unusable; otherwise, set to 1. 
— TR. Set to 0. 


The host-state area does not contain a selector field for LDTR. LDTR is established as follows on all VM exits: the 
selector is cleared to OOOOH, the segment is marked unusable and is otherwise undefined (although the base 
address is always canonical). 


The base addresses for GDTR and IDTR are loaded from the GDTR base-address field and the IDTR base-address 
field, respectively. If the processor supports the Intel 64 architecture and the processor supports N < 64 linear- 
address bits, each of bits 63:N of each base address is set to the value of bit N-1 of that base address. The GDTR 
and IDTR limits are each set to FFFFH. 


. Software can determine the number N by executing CPUID with 8OOOO0O08H in EAX. The number of linear-address bits supported is 
returned in bits 15:8 of EAX. 
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26.5.3 Loading Host RIP, RSP, RFLAGS, and SSP 


RIP and RSP are loaded from the RIP field and the RSP field, respectively. RFLAGS is cleared, except bit 1, which is 
always set. 


If the “load CET” VM-exit control is 1, SSP (shadow-stack pointer) is loaded from the SSP field. 


26.5.4 Checking and Loading Host Page-Directory-Pointer-Table Entries 


If CRO.PG = 1, CR4.PAE = 1, and IA32_EFER.LMA = 0, the logical processor uses PAE paging. See Section 4.4 of 
the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A.' When in PAE paging is in use, the 
physical address in CR3 references a table of page-directory-pointer-table entries (PDPTEs). A MOV to CR3 
when PAE paging is in use checks the validity of the PDPTEs and, if they are valid, loads them into the processor 
(into internal, non-architectural registers). 


AVM exit is to a VMM that uses PAE paging if (1) bit 5 (corresponding to CR4.PAE) is set in the CR4 field in the host- 
state area of the VMCS; and (2) the “host address-space size” VM-exit control is 0. Such a VM exit may check the 
validity of the PDPTEs referenced by the CR3 field in the host-state area of the VMCS. Such a VM exit must check 
their validity if either (1) PAE paging was not in use before the VM exit; or (2) the value of CR3 is changing as a 
result of the VM exit. A VM exit to a VMM that does not use PAE paging must not check the validity of the PDPTEs. 


A VM exit that checks the validity of the PDPTEs uses the same checks that are used when CR3 is loaded with 
MOV to CR3 when PAE paging is in use. If MOV to CR3 would cause a general-protection exception due to the 
PDPTEs that would be loaded (e.g., because a reserved bit is set), a VMX abort occurs (see Section 26.7). Ifa 
VM exit to a VMM that uses PAE does not cause a VMX abort, the PDPTEs are loaded into the processor as would 
MOV to CR3, using the value of CR3 being load by the VM exit. 


26.5.5 Updating Non-Register State 
VM exits affect the non-register state of a logical processor as follows: 
¢ A logical processor is always in the active state after a VM exit. 
¢ Event blocking is affected as follows: 
— There is no blocking by STI or by MOV SS after a VM exit. 


— VM exits caused directly by non-maskable interrupts (NMIs) cause blocking by NMI (see Table 23-3). Other 
VM exits do not affect blocking by NMI. (See Section 26.1 for the case in which an NMI causes a VM exit 
indirectly.) 

¢ There are no pending debug exceptions after a VM exit. 


Section 27.3 describes how the VMX architecture controls how a logical processor manages information in the TLBs 
and paging-structure caches. The following items detail how VM exits invalidate cached mappings: 


e If the “enable VPID” VM-execution control is 0, the logical processor invalidates linear mappings and combined 
mappings associated with VPID OOOOH (for all PCIDs); combined mappings for VPID OOOOH are invalidated for 
all EP4TA values (EP4TA is the value of bits 51:12 of EPTP). 


e VM exits are not required to invalidate any guest-physical mappings, nor are they required to invalidate any 
linear mappings or combined mappings if the “enable VPID” VM-execution control is 1. 


26.5.6 Clearing Address-Range Monitoring 


The Intel 64 and IA-32 architectures allow software to monitor a specified address range using the MONITOR and 
MWAIT instructions. See Section 8.10.4 in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 3A. VM exits clear any address-range monitoring that may be in effect. 


1. On processors that support Intel 64 architecture, the physical-address extension may support more than 36 physical-address bits. 
Software can determine a processor's physical-address width by executing CPUID with 80000008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 
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26.6 LOADING MSRS 


VM exits may load MSRs from the VM-exit MSR-load area (see Section 23.7.2). Specifically each entry in that area 
(up to the number specified in the VM-exit MSR-load count) is processed in order by loading the MSR indexed by 
bits 31:0 with the contents of bits 127:64 as they would be written by WRMSR. 


Processing of an entry fails in any of the following cases: 


* The value of bits 31:0 is either CO000100H (the IA32_FS_BASE MSR) or C0000101H (the IA32_GS_BASE 
MSR). 


° The value of bits 31:8 is OO0008H, meaning that the indexed MSR is one that allows access to an APIC register 
when the local APIC is in x2APIC mode. 


° The value of bits 31:0 indicates an MSR that can be written only in system-management mode (SMM) and the 
VM exit will not end in SMM. (IA32_SMM_MONITOR_CTL is an MSR that can be written only in SMM.) 


¢ = The value of bits 31:0 indicates an MSR that cannot be loaded on VM exits for model-specific reasons. A 
processor may prevent loading of certain MSRs even if they can normally be written by WRMSR. Such model- 
specific behavior is documented in Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 4. 


¢  =Bits 63:32 are not all 0. 


¢ Anattempt to write bits 127:64 to the MSR indexed by bits 31:0 of the entry would cause a general-protection 
exception if executed via WRMSR with CPL = 0.! 


If processing fails for any entry, a VMX abort occurs. See Section 26.7. 


If any MSR is being loaded in such a way that would architecturally require a TLB flush, the TLBs are updated so 
that, after VM exit, the logical processor does not use any translations that were cached before the transition. 


26.7. VMX ABORTS 


A problem encountered during a VM exit leads to a VMX abort. A VMX abort takes a logical processor into a shut- 
down state as described below. 


A VMX abort does not modify the VMCS data in the VMCS region of any active VMCS. The contents of these data 
are thus suspect after the VMX abort. 


On a VMX abort, a logical processor saves a nonzero 32-bit VMX-abort indicator field at byte offset 4 in the VMCS 
region of the VMCS whose misconfiguration caused the failure (see Section 23.2). The following values are used: 


1. There was a failure in saving guest MSRs (see Section 26.4). 
2. Host checking of the page-directory-pointer-table entries (PDPTEs) failed (see Section 26.5.4). 


3. The current VMCS has been corrupted (through writes to the corresponding VMCS region) in such a way that 
the logical processor cannot complete the VM exit properly. 


There was a failure on loading host MSRs (see Section 26.6). 
There was a machine-check event during VM exit (see Section 26.8). 


The logical processor was in IA-32e mode before the VM exit and the “host address-space size” VM-exit control 
was 0 (see Section 26.5). 


Some of these causes correspond to failures during the loading of state from the host-state area. Because the 
loading of such state may be done in any order (see Section 26.5) a VM exit that might lead to a VMX abort for 
multiple reasons (for example, the current VMCS may be corrupt and the host PDPTEs might not be properly 
configured). In such cases, the VMX-abort indicator could correspond to any one of those reasons. 


A logical processor never reads the VMX-abort indicator in a VMCS region and writes it only with one of the non- 
zero values mentioned above. The VMX-abort indicator allows software on one logical processor to diagnose the 


1. Note the following about processors that support Intel 64 architecture. If CRO.PG = 1, WRMSR to the IA32_EFER MSR causes a gen- 
eral-protection exception if it would modify the LME bit. Since CRO.PG is always 1 in VMX operation, the IA32_EFER MSR should not 
be included in the VM-exit MSR-load area for the purpose of modifying the LME bit. 
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VMxX-abort on another. For this reason, it is recommended that software running in VMX root operation zero the 
VMxX-abort indicator in the VMCS region of any VMCS that it uses. 


After saving the VMX-abort indicator, operation of a logical processor experiencing a VMX abort depends on 
whether the logical processor is in SMX operation:! 


* If the logical processor is in SMX operation, an Intel® TXT shutdown condition occurs. The error code used is 
OOODH, indicating “VMX abort.” See Inte/® Trusted Execution Technology Measured Launched Environment 
Programming Guide. 


e If the logical processor is outside SMX operation, it issues a special bus cycle (to notify the chipset) and enters 
the VMX-abort shutdown state. RESET is the only event that wakes a logical processor from the VMX-abort 
shutdown state. The following events do not affect a logical processor in this state: machine-check events; INIT 
signals; external interrupts; non-maskable interrupts (NMIs); start-up IPIs (SIPIs); and system-management 
interrupts (SMIs). 


26.8 MACHINE-CHECK EVENTS DURING VM EXIT 


If a machine-check event occurs during VM exit, one of the following occurs: 
¢ The machine-check event is handled as if it occurred before the VM exit: 


— if eee = 0, operation of the logical processor depends on whether the logical processor is in SMX 
operation: 


* If the logical processor is in SMX operation, an Intel® TXT shutdown condition occurs. The error code 
used is OOOCH, indicating “unrecoverable machine-check condition.” 


¢ If the logical processor is outside SMX operation, it goes to the shutdown state. 
— If CR4.MCE = 1, a machine-check exception (#MC) is generated: 
¢ If bit 18 (#MC) of the exception bitmap is 0, the exception is delivered through the guest IDT. 
¢ If bit 18 of the exception bitmap is 1, the exception causes a VM exit. 
¢ The machine-check event is handled after VM exit completes: 


— Ifthe VM exit ends with CR4.MCE = 0, operation of the logical processor depends on whether the logical 
processor is in SMX operation: 


* If the logical processor is in SMX operation, an Intel® TXT shutdown condition occurs with error code 
OOOCH (unrecoverable machine-check condition). 


¢ If the logical processor is outside SMX operation, it goes to the shutdown state. 
— Ifthe VM exit ends with CR4.MCE = 1, a machine-check exception (#MC) is delivered through the host IDT. 


¢ AVMxX abort is generated (see Section 26.7). The logical processor blocks events as done normally in 
VMX abort. The VMX abort indicator is 5, for “machine-check event during VM exit.” 


The first option is not used if the machine-check event occurs after any host state has been loaded. The second 
option is used only if VM entry is able to load all host state. 


1. A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. A logi- 
cal processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the last 
execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference,” in Intel® 64 and IA-32 Architectures Software 
Developer's Manual, Volume 2B. 


2. A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. A logi- 
cal processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the last 


execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference,” in Intel? 64 and IA-32 Architectures Software 
Developer's Manual, Volume 2B. 
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VMX SUPPORT FOR ADDRESS TRANSLATION 


The architecture for VMX operation includes two features that support address translation: virtual-processor iden- 
tifiers (VPIDs) and the extended page-table mechanism (EPT). VPIDs are a mechanism for managing translations 
of linear addresses. EPT defines a layer of address translation that augments the translation of linear addresses. 


Section 27.1 details the architecture of VPIDs. Section 27.2 provides the details of EPT. Section 27.3 explains how 
a logical processor may cache information from the paging structures, how it may use that cached information, and 
how software can managed the cached information. 


27.1 VIRTUAL PROCESSOR IDENTIFIERS (VPIDS) 


The original architecture for VMX operation required VMX transitions to flush the TLBs and paging-structure caches. 
This ensured that translations cached for the old linear-address space would not be used after the transition. 


Virtual-processor identifiers (VPIDs) introduce to VMX operation a facility by which a logical processor may cache 
information for multiple linear-address spaces. When VPIDs are used, VMX transitions may retain cached informa- 
tion and the logical processor switches to a different linear-address space. 


Section 27.3 details the mechanisms by which a logical processor manages information cached for multiple address 
spaces. A logical processor may tag some cached information with a 16-bit VPID. This section specifies how the 
current VPID is determined at any point in time: 


¢ The current VPID is OOOOH in the following situations: 


— Outside VMX operation. (This includes operation in system-management mode under the default treatment 
of SMIs and SMM with VMX operation; see Section 30.14.) 


— In VMX root operation. 
— In VMX non-root operation when the “enable VPID” VM-execution control is 0. 


¢ If the logical processor is in VMX non-root operation and the “enable VPID” VM-execution control is 1, the 
current VPID is the value of the VPID VM-execution control field in the VMCS. (VM entry ensures that this value 
is never OOOOH; see Section 25.2.1.1.) 


VPIDs and PCIDs (see Section 4.10.1) can be used concurrently. When this is done, the processor associates 
cached information with both a VPID and a PCID. Such information is used only if the current VPID and PCID both 
match those associated with the cached information. 


27.2 THE EXTENDED PAGE TABLE MECHANISM (EPT) 


The extended page-table mechanism (EPT) is a feature that can be used to support the virtualization of physical 
memory. When EPT is in use, certain addresses that would normally be treated as physical addresses (and used to 
access memory) are instead treated as guest-physical addresses. Guest-physical addresses are translated by 
traversing a set of EPT paging structures to produce physical addresses that are used to access memory. 


* Section 27.2.1 gives an overview of EPT. 

® Section 27.2.2 describes operation of EPT-based address translation. 

* Section 27.2.3 discusses VM exits that may be caused by EPT. 

° Section 27.2.7 describes interactions between EPT and memory typing. 


27.2.1 EPT Overview 


EPT is used when the “enable EPT” VM-execution control is 1.! It translates the guest-physical addresses used in 
VMX non-root operation and those used by VM entry for event injection. 
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The translation from guest-physical addresses to physical addresses is determined by a set of EPT paging struc- 
tures. The EPT paging structures are similar to those used to translate linear addresses while the processor is in 
IA-32e mode. Section 27.2.2 gives the details of the EPT paging structures. 


If CRO.PG = 1, linear addresses are translated through paging structures referenced through control register CR3. 
While the “enable EPT” VM-execution control is 1, these are called guest paging structures. There are no guest 
paging structures if CRO.PG = 0.1 


When the “enable EPT” VM-execution control is 1, the identity of guest-physical addresses depends on the value 
of CRO.PG: 


¢ If CRO.PG = 0, each linear address is treated as a guest-physical address. 


°¢ If CRO.PG = 1, guest-physical addresses are those derived from the contents of control register CR3 and the 
guest paging structures. (This includes the values of the PDPTEs, which logical processors store in internal, 
non-architectural registers.) The latter includes (in page-table entries and in other paging-structure entries for 
which bit 7—PS—is 1) the addresses to which linear addresses are translated by the guest paging structures. 


If CRO.PG = 1, the translation of a linear address to a physical address requires multiple translations of guest-phys- 
ical addresses using EPT. Assume, for example, that CR4.PAE = CR4.PSE = 0. The translation of a 32-bit linear 
address then operates as follows: 


° Bits 31:22 of the linear address select an entry in the guest page directory located at the guest-physical 
address in CR3. The guest-physical address of the guest page-directory entry (PDE) is translated through EPT 
to determine the guest PDE’s physical address. 


° Bits 21:12 of the linear address select an entry in the guest page table located at the guest-physical address in 
the guest PDE. The guest-physical address of the guest page-table entry (PTE) is translated through EPT to 
determine the guest PTE’s physical address. 


¢ Bits 11:0 of the linear address is the offset in the page frame located at the guest-physical address in the guest 
PTE. The guest-physical address determined by this offset is translated through EPT to determine the physical 
address to which the original linear address translates. 


In addition to translating a guest-physical address to a physical address, EPT specifies the privileges that software 
is allowed when accessing the address. Attempts at disallowed accesses are called EPT violations and cause 
VM exits. See Section 27.2.3. 


A processor uses EPT to translate guest-physical addresses only when those addresses are used to access memory. 

This principle implies the following: 

¢ The MOV to CR3 instruction loads CR3 with a guest-physical address. Whether that address is translated 
through EPT depends on whether PAE paging is being used.2 


— If PAE paging is not being used, the instruction does not use that address to access memory and does not 
cause it to be translated through EPT. (If CRO.PG = 1, the address will be translated through EPT on the 
next memory accessing using a linear address.) 


— If PAE paging is being used, the instruction loads the four (4) page-directory-pointer-table entries (PDPTEs) 
from that address and it does cause the address to be translated through EPT. 


° Section 4.4.1 identifies executions of MOV to CRO and MOV to CR4 that load the PDPTEs from the guest- 
physical address in CR3. Such executions cause that address to be translated through EPT. 


¢ The PDPTEs contain guest-physical addresses. The instructions that load the PDPTEs (see above) do not use 
those addresses to access memory and do not cause them to be translated through EPT. The address in a 
PDPTE will be translated through EPT on the next memory accessing using a linear address that uses that 
PDPTE. 


1. “Enable EPT” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution controls 
is 0, the logical processor operates as if the “enable EPT” VM-execution control were 0. See Section 23.6.2. 

1. If the capability MSR 1A32_VMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, CRO.PG can be 0 in VMX non-root 
operation only if the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are 
both 1. 

2. A logical processor uses PAE paging if CRO.PG = 1, CR4.PAE = 1 and IA32_EFER.LMA = 0. See Section 4.4 in the Intel® 64 and IA-32 
Architectures Software Developer's Manual, Volume 3A. 
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EPT Translation Mechanism 


The EPT translation mechanism uses only bits 47:0 of each guest-physical address.! It uses a page-walk length of 


4, meaning that at most 4 EPT paging-structure entries are accessed to translate a guest-physical address. 


2 


These 48 bits are partitioned by the logical processor to traverse the EPT paging structures: 


¢ A4-KByte naturally aligned EPT PML4 table is located at the physical address specified in bits 51:12 of the 
extended-page-table pointer (EPTP), a VM-execution control field (see Table 23-9 in Section 23.6.11). An EPT 
PML4 table comprises 512 64-bit entries (EPT PML4Es). An EPT PML4E is selected using the physical address 
defined as follows: 


— Bits 63:52 are all 0. 

— Bits 51:12 are from the EPTP. 

— Bits 11:3 are bits 47:39 of the guest-physical address. 
— Bits 2:0 are all 0. 


Because an EPT PML4E is identified using bits 47:39 of the guest-physical address, it controls access to a 512- 
GByte region of the guest-physical-address space. The format of an EPT PML4E is given in Table 27-1. 


Table 27-1. Format of an EPT PML4 Entry (PML4E) that References an EPT Page-Directory-Pointer Table 


Bit Contents 

Position(s) 

0 Read access; indicates whether reads are allowed from the 512-GByte region controlled by this entry 

1 Write access; indicates whether writes are allowed to the 512-GByte region controlled by this entry 

2 If the “mode-based execute control for EPT” VM-execution control is 0, execute access; indicates whether instruction 
fetches are allowed from the 512-GByte region controlled by this entry 
If that control is 1, execute access for supervisor-mode linear addresses; indicates whether instruction fetches are 
allowed from supervisor-mode linear addresses in the 512-GByte region controlled by this entry 

7:3 Reserved (must be 0) 

8 If bit 6 of EPTP is 1, accessed flag for EPT; indicates whether software has accessed the 512-GByte region 
controlled by this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 

9 Ignored 

10 Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 
1, indicates whether instruction fetches are allowed from user-mode linear addresses in the 512-GByte region 
controlled by this entry. If that control is 0, this bit is ignored. 

11 Ignored 

(N-1):12 Physical address of 4-KByte aligned EPT page-directory-pointer table referenced by this entry! 

51:N Reserved (must be 0) 

63:52 Ignored 


1. No processors supporting the Intel 64 architecture support more than 48 physical-address bits. Thus, no such processor can pro- 
duce a quest-physical address with more than 48 bits. An attempt to use such an address causes a page fault. An attempt to load 
CR3 with such an address causes a general-protection fault. If PAE paging is being used, an attempt to load CR3 that would load a 
PDPTE with such an address causes a general-protection fault. 


2. Future processors may include support for other EPT page-walk lengths. Software should read the VMX capability MSR 
IA32_VMX_EPT_VPID_CAP (see Appendix A.10) to determine what EPT page-walk lengths are supported. 
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NOTES: 
1. Nis the physical-address width supported by the processor. Software can determine a processor's physical-address width by execut- 
ing CPUID with BOOOO0O8H in EAX. The physical-address width is returned in bits 7:0 of EAX. 


¢ A4-KByte naturally aligned EPT page-directory-pointer table is located at the physical address specified in 
bits 51:12 of the EPT PML4E. An EPT page-directory-pointer table comprises 512 64-bit entries (EPT PDPTEs). 
An EPT PDPTE is selected using the physical address defined as follows: 


— Bits 63:52 are all 0. 

— Bits 51:12 are from the EPT PML4E. 

— Bits 11:3 are bits 38:30 of the guest-physical address. 
— Bits 2:0 areall 0. 


Because an EPT PDPTE is identified using bits 47:30 of the guest-physical address, it controls access to a 1-GByte 
region of the guest-physical-address space. Use of the EPT PDPTE depends on the value of bit 7 in that entry:! 


e If bit 7 of the EPT PDPTE is 1, the EPT PDPTE maps a 1-GByte page. The final physical address is computed as 
follows: 


— Bits 63:52 are all 0. 

— Bits 51:30 are from the EPT PDPTE. 

— Bits 29:0 are from the original guest-physical address. 

The format of an EPT PDPTE that maps a 1-GByte page is given in Table 27-2. 


e If bit 7 of the EPT PDPTE is O, a 4-KByte naturally aligned EPT page directory is located at the physical address 
specified in bits 51:12 of the EPT PDPTE. The format of an EPT PDPTE that references an EPT page directory is 
given in Table 27-3. 


1. Not all processors allow bit 7 of an EPT PDPTE to be set to 1. Software should read the VMX capability MSR 
IA32_V/MX_EPT_\VPID_CAP (see Appendix A.10) to determine whether this is allowed. 
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Table 27-2. Format of an EPT Page-Directory-Pointer-Table Entry (PDPTE) that Maps a 1-GByte Page 


Bit Contents 

Position(s) 

0 Read access; indicates whether reads are allowed from the 1-GByte page referenced by this entry 

1 Write access; indicates whether writes are allowed to the 1-GByte page referenced by this entry 

2 If the “mode-based execute control for EPT” VM-execution control is 0, execute access; indicates whether 


instruction fetches are allowed from the 1-GByte page controlled by this entry 


If that control is 1, execute access for supervisor-mode linear addresses; indicates whether instruction fetches are 
allowed from supervisor-mode linear addresses in the 1-GByte page controlled by this entry 


5:3 EPT memory type for this 1-GByte page (see Section 27.2.7) 

6 Ignore PAT memory type for this 1-GByte page (see Section 27.2.7) 

7 Must be 1 (otherwise, this entry references an EPT page directory) 

8 If bit 6 of EPTP is 1, accessed flag for EPT; indicates whether software has accessed the 1-GByte page referenced 


by this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 


9 If bit 6 of EPTP is 1, dirty flag for EPT; indicates whether software has written to the 1-GByte page referenced by 
this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 


10 Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 
1, indicates whether instruction fetches are allowed from user-mode linear addresses in the 1-GByte page controlled 
by this entry. If that control is 0, this bit is ignored. 


11 Ignored 


29:12 Reserved (must be 0) 


(N-1):30 Physical address of the 1-GByte page referenced by this entry! 


51:N Reserved (must be 0) 
59:52 Ignored 
60 Supervisor shadow stack. If bit 7 of EPTP is 1, indicates whether supervisor shadow stack accesses are allowed to 


guest-physical addresses in the 1-GByte page mapped by this entry (see Section 27.2.3.2). 
Ignored if bit 7 of EPTP is O 


62:61 Ignored 


63 Suppress #VE. If the “EPT-violation #VE"” VM-execution control is 1, EPT violations caused by accesses to this page 
are convertible to virtualization exceptions only if this bit is O (see Section 24.5.7.1). If “EPT-violation #VE" VM- 
execution control is 0, this bit is ignored. 


NOTES: 
1. N is the physical-address width supported by the logical processor. 
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Table 27-3. Format of an EPT Page-Directory-Pointer-Table Entry (PDPTE) that References an EPT Page Directory 


Bit Contents 

Position(s) 

0 Read access; indicates whether reads are allowed from the 1-GByte region controlled by this entry 

1 Write access; indicates whether writes are allowed to the 1-GByte region controlled by this entry 

2 If the “mode-based execute control for EPT” VM-execution control is 0, execute access; indicates whether instruction 
fetches are allowed from the 1-GByte region controlled by this entry 
If that control is 1, execute access for supervisor-mode linear addresses; indicates whether instruction fetches are 
allowed from supervisor-mode linear addresses in the 1-GByte region controlled by this entry 

7:3 Reserved (must be 0) 

8 If bit 6 of EPTP is 1, accessed flag for EPT; indicates whether software has accessed the 1-GByte region controlled 
by this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 

9 Ignored 

10 Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 
1, indicates whether instruction fetches are allowed from user-mode linear addresses in the 1-GByte region 
controlled by this entry. If that control is 0, this bit is ignored. 

11 Ignored 

(N-1):12 Physical address of 4-KByte aligned EPT page directory referenced by this entry! 

51:N Reserved (must be 0) 

63:52 Ignored 

NOTES: 


1. N is the physical-address width supported by the logical processor. 


An EPT page-directory comprises 512 64-bit entries (PDEs). An EPT PDE is selected using the physical address 
defined as follows: 


— Bits 63:52 are all 0. 

— Bits 51:12 are from the EPT PDPTE. 

— Bits 11:3 are bits 29:21 of the guest-physical address. 
— Bits 2:0 areall 0. 


Because an EPT PDE is identified using bits 47:21 of the guest-physical address, it controls access to a 2-MByte 
region of the guest-physical-address space. Use of the EPT PDE depends on the value of bit 7 in that entry: 


° If bit 7 of the EPT PDE is 1, the EPT PDE maps a 2-MByte page. The final physical address is computed as 


follows: 


— Bits 63:52 are all 0. 
— Bits 51:21 are from the EPT PDE. 
— Bits 20:0 are from the original guest-physical address. 
The format of an EPT PDE that maps a 2-MByte page is given in Table 27-4. 
e If bit 7 of the EPT PDE is 0, a 4-KByte naturally aligned EPT page table is located at the physical address 


specified in bits 51:12 of the EPT PDE. The format of an EPT PDE that references an EPT page table is given in 
Table 27-5. 


An EPT page table comprises 512 64-bit entries (PTEs). An EPT PTE is selected using a physical address defined 
as follows: 


— Bits 63:52 are all 0. 


27-6 Vol. 3C 


VMX SUPPORT FOR ADDRESS TRANSLATION 


Table 27-4. Format of an EPT Page-Directory Entry (PDE) that Maps a 2-MByte Page 


Bit Contents 

Position(s) 

0 Read access; indicates whether reads are allowed from the 2-MByte page referenced by this entry 

1 Write access; indicates whether writes are allowed to the 2-MByte page referenced by this entry 

2 If the “mode-based execute control for EPT” VM-execution control is 0, execute access; indicates whether instruction 


fetches are allowed from the 2-MByte page controlled by this entry 


If that control is 1, execute access for supervisor-mode linear addresses; indicates whether instruction fetches are 
allowed from supervisor-mode linear addresses in the 2-MByte page controlled by this entry 


5:3 EPT memory type for this 2-MByte page (see Section 27.2.7) 

6 Ignore PAT memory type for this 2-MByte page (see Section 27.2.7) 

7 Must be 1 (otherwise, this entry references an EPT page table) 

8 If bit 6 of EPTP is 1, accessed flag for EPT; indicates whether software has accessed the 2-MByte page referenced 


by this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 


9 If bit 6 of EPTP is 1, dirty flag for EPT; indicates whether software has written to the 2-MByte page referenced by 
this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 


10 Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 
1, indicates whether instruction fetches are allowed from user-mode linear addresses in the 2-MByte page controlled 
by this entry. If that control is 0, this bit is ignored. 


11 Ignored 

20:12 Reserved (must be 0) 

(N-1):21 Physical address of the 2-MByte page referenced by this entry! 

51:N Reserved (must be 0) 

59:52 Ignored 

60 Supervisor shadow stack. If bit 7 of EPTP is 1, indicates whether supervisor shadow stack accesses are allowed to 


guest-physical addresses in the 2-MByte page mapped by this entry (see Section 27.2.3.2). 
Ignored if bit 7 of EPTP is O 


62:61 Ignored 


63 Suppress #VE. If the “EPT-violation #VE" VM-execution control is 1, EPT violations caused by accesses to this page 
are convertible to virtualization exceptions only if this bit is O (see Section 24.5.7.1). If “EPT-violation #VE” VM- 
execution control is 0, this bit is ignored. 


NOTES: 
1. Nis the physical-address width supported by the logical processor. 


— Bits 51:12 are from the EPT PDE. 
— Bits 11:3 are bits 20:12 of the guest-physical address. 
— Bits 2:0 areall 0. 


¢ Because an EPT PTE is identified using bits 47:12 of the guest-physical address, every EPT PTE maps a 4-KByte 
page. The final physical address is computed as follows: 


— Bits 63:52 are all 0. 
— Bits 51:12 are from the EPT PTE. 
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— Bits 11:0 are from the original guest-physical address. 
The format of an EPT PTE is given in Table 27-6. 


An EPT paging-structure entry is present if any of bits 2:0 is 1; otherwise, the entry is not present. The processor 
ignores bits 62:3 and uses the entry neither to reference another EPT paging-structure entry nor to produce a 
physical address. A reference using a guest-physical address whose translation encounters an EPT paging-struc- 
ture that is not present causes an EPT violation (see Section 27.2.3.2). (If the “EPT-violation #VE” VM-execution 
control is 1, the EPT violation is convertible to a virtualization exception only if bit 63 is 0; see Section 24.5.7.1. If 
the “EPT-violation #VE” VM-execution control is 0, this bit is ignored.) 


Table 27-5. Format of an EPT Page-Directory Entry (PDE) that References an EPT Page Table 


Bit Contents 

Position(s) 

0 Read access; indicates whether reads are allowed from the 2-MByte region controlled by this entry 

1 Write access; indicates whether writes are allowed to the 2-MByte region controlled by this entry 

2 If the “mode-based execute control for EPT” VM-execution control is 0, execute access; indicates whether instruction 


fetches are allowed from the 2-MByte region controlled by this entry 


If that control is 1, execute access for supervisor-mode linear addresses; indicates whether instruction fetches are 
allowed from supervisor-mode linear addresses in the 2-MByte region controlled by this entry 


6:3 Reserved (must be 0) 

7 Must be 0 (otherwise, this entry maps a 2-MByte page) 

8 If bit 6 of EPTP is 1, accessed flag for EPT; indicates whether software has accessed the 2-MByte region controlled 
by this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 

9 Ignored 

10 Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 


1, indicates whether instruction fetches are allowed from user-mode linear addresses in the 2-MByte region 
controlled by this entry. If that control is 0, this bit is ignored. 


11 Ignored 


(N-1):12 Physical address of 4-KByte aligned EPT page table referenced by this entry! 


51:N Reserved (must be 0) 
63:52 Ignored 
NOTES: 


1. Nis the physical-address width supported by the logical processor. 


NOTE 


If the “mode-based execute control for EPT” VM-execution control is 1, an EPT paging-structure 
entry is present if any of bits 2:0 or bit 10 is 1. If bits 2:0 are all 0 but bit 10 is 1, the entry is used 
normally to reference another EPT paging-structure entry or to produce a physical address. 


The discussion above describes how the EPT paging structures reference each other and how the logical processor 
traverses those structures when translating a guest-physical address. It does not cover all details of the translation 
process. Additional details are provided as follows: 


® Situations in which the translation process may lead to VM exits (sometimes before the process completes) are 
described in Section 27.2.3. 


¢ Interactions between the EPT translation mechanism and memory typing are described in Section 27.2.7. 


Figure 27-1 gives a summary of the formats of the EPTP and the EPT paging-structure entries. For the EPT paging 
structure entries, it identifies separately the format of entries that map pages, those that reference other EPT 
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paging structures, and those that do neither because they are not present; bits 2:0 and bit 7 are highlighted 
because they determine how a paging-structure entry is used. (Figure 27-1 does not comprehend the fact that, if 
the “mode-based execute control for EPT” VM-execution control is 1, an entry is present if any of bits 2:0 or bit 10 
is 1.) 


27.2.3 EPT-Induced VM Exits 


Accesses using guest-physical addresses may cause VM exits due to EPT misconfigurations, EPT violations, and 
page-modification log-full events. An EPT misconfiguration occurs when, in the course of translating a guest- 
physical address, the logical processor encounters an EPT paging-structure entry that contains an unsupported 
value (see Section 27.2.3.1). An EPT violation occurs when there is no EPT misconfiguration but the EPT paging- 
structure entries disallow an access using the guest-physical address (see Section 27.2.3.2). A page-modifica- 
tion log-full event occurs when the logical processor determines a need to create a page-modification log entry 
and the current log is full (see Section 27.2.6). 


These events occur only due to an attempt to access memory with a guest-physical address. Loading CR3 with a 
guest-physical address with the MOV to CR3 instruction can cause neither an EPT configuration nor an EPT violation 
until that address is used to access a paging structure.! 


If the “EPT-violation #VE” VM-execution control is 1, certain EPT violations may cause virtualization exceptions 
instead of VM exits. See Section 24.5.7.1. 


27.2.3.1_ EPT Misconfigurations 


An EPT misconfiguration occurs if translation of a guest-physical address encounters an EPT paging-structure that 

meets any of the following conditions: 

¢ Bit 0 of the entry is clear (indicating that data reads are not allowed) and bit 1 is set (indicating that data writes 
are allowed). 


e Either of the following if the processor does not support execute-only translations: 

— Bit 0 of the entry is clear (indicating that data reads are not allowed) and bit 2 is set (indicating that 
instruction fetches are allowed). 

— The “mode-based execute control for EPT” VM-execution control is 1, bit 0 of the entry is clear (indicating 
that data reads are not allowed), and bit 10 is set (indicating that instruction fetches are allowed from user- 
mode linear addresses). 

Software should read the VMX capability MSR IA32_VMX_EPT_VPID_CAP to determine whether execute-only 

translations are supported (see Appendix A.10). 

¢ The entry is present (see Section 27.2.2) and one of the following holds: 

— Areserved bit is set. This includes the setting of a bit in the range 51:12 that is beyond the logical 
processor's physical-address width.? See Section 27.2.2 for details of which bits are reserved in which EPT 
paging-structure entries. 


— The entry is the last one used to translate a guest physical address (either an EPT PDE with bit 7 set to 1 or 
an EPT PTE) and the value of bits 5:3 (EPT memory type) is 2, 3, or 7 (these values are reserved). 


EPT misconfigurations result when an EPT paging-structure entry is configured with settings reserved for future 
functionality. Software developers should be aware that such settings may be used in the future and that an EPT 
paging-structure entry that causes an EPT misconfiguration on one processor might not do so in the future. 


1. If the logical processor is using PAE paging—because CRO.PG = CR4.PAE = 1 and IA32_EFER.LMA = O—the MOV to CR3 instruction 
loads the PDPTEs from memory using the guest-physical address being loaded into CR3. In this case, therefore, the MOV to CR3 
instruction may cause an EPT misconfiguration, an EPT violation, or a page-modification log-full event. 

2. If the “mode-based execute control for EPT” VM-execution control is 1, setting bit 2 indicates that instruction fetches are allowed 
from supervisor-mode linear addresses. 

3. Software can determine a processor's physical-address width by executing CPUID with BOOOOOO8H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 
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Table 27-6. Format of an EPT Page-Table Entry that Maps a 4-KByte Page 


Bit Contents 

Position(s) 

0 Read access; indicates whether reads are allowed from the 4-KByte page referenced by this entry 

1 Write access; indicates whether writes are allowed to the 4-KByte page referenced by this entry 

2 If the “mode-based execute control for EPT” VM-execution control is 0, execute access; indicates whether 


instruction fetches are allowed from the 4-KByte page controlled by this entry 


If that control is 1, execute access for supervisor-mode linear addresses; indicates whether instruction fetches are 
allowed from supervisor-mode linear addresses in the 4-KByte page controlled by this entry 


5:3 EPT memory type for this 4-KByte page (see Section 27.2.7) 

6 Ignore PAT memory type for this 4-KByte page (see Section 27.2.7) 

7 Ignored 

8 If bit 6 of EPTP is 1, accessed flag for EPT; indicates whether software has accessed the 4-KByte page referenced 


by this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 


9 If bit 6 of EPTP is 1, dirty flag for EPT; indicates whether software has written to the 4-KByte page referenced by 
this entry (see Section 27.2.5). Ignored if bit 6 of EPTP is O 


10 Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 
1, indicates whether instruction fetches are allowed from user-mode linear addresses in the 4-KByte page controlled 
by this entry. If that control is 0, this bit is ignored. 


11 Ignored 


(N-1):12 Physical address of the 4-KByte page referenced by this entry! 


51:N Reserved (must be 0) 
59:52 Ignored 
60 Supervisor shadow stack. If bit 7 of EPTP is 1, indicates whether supervisor shadow stack accesses are allowed to 


guest-physical addresses in the 4-KByte page mapped by this entry (see Section 27.2.3.2). 
Ignored if bit 7 of EPTP is O 


61 Sub-page write permissions. If the “sub-page write permissions for EPT” VM-execution control is 1, writes to 
individual 128-byte regions of the 4-KByte page referenced by this entry may be allowed even if the page would 
normally not be writable (see Section 27.2.4). If “sub-page write permissions for EPT” VM-execution control is 0, this 


bit is ignored. 
62 Ignored 
63 Suppress #VE. If the “EPT-violation #VE" VM-execution control is 1, EPT violations caused by accesses to this page 


are convertible to virtualization exceptions only if this bit is O (see Section 24.5.7.1). If “EPT-violation #VE” VM- 
execution control is 0, this bit is ignored. 


NOTES: 
1. Nis the physical-address width supported by the logical processor. 


27.2.3.2 €EPT Violations 


An EPT violation may occur during an access using a guest-physical address whose translation does not cause an 
EPT misconfiguration. An EPT violation occurs in any of the following situations: 


° Translation of the guest-physical address encounters an EPT paging-structure entry that is not present (see 
Section 27.2.2). 
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¢ The access is a data read and, for any byte to be read, bit 0 (read access) was clear in any of the EPT paging- 
structure entries used to translate the guest-physical address of the byte. Reads by the logical processor of 
guest paging structures to translate a linear address are considered to be data reads. 


Reserved Address of EPT PML4 table 


R PML4E; 
present 


Ignored ; Address of EPT page-directory-pointer table Reserved 


PML4E: 
Ignored not 
present 


Physical PDPTE: 
Ignored F address of Reserved U 1GB 
1GB page , page 


PDPTE: 
Ignored . Address of EPT page directory 10) page 
lle directory 


PDTPE: 
Ignored not 
present 


: PDE: 
Physical address 
Ignored . of 2MB page Reserved 2MB 


page 


PDE: 
Ignored i Address of EPT page table U (0) page 
saa table 


PDE: 
Ignored not 
present 


PTE: 
Ignored F Physical address of 4KB page 4KB 
; page 


PTE: 
Ignored not 
present 


mounmaounman 


Figure 27-1. Formats of EPTP and EPT Paging-Structure Entries 


NOTES: 

1.Mis an abbreviation for MAXPHYADDR. 

2. Supervisor shadow-stack control. 

3. See Section 23.6.11 for details of the EPTP. 


4. Execute access for user-mode linear addresses. If the “mode-based execute control for EPT” VM-execution control is 0, this bit is 
ignored. 


5. Execute access. If the “mode-based execute control for EPT” VM-execution control is 1, this bit controls execute access for supervi- 
sor-mode linear addresses. 


6. If the “mode-based execute control for EPT” VM-execution control is 1, an EPT paging-structure entry is present if any of bits 2:0 or 
bit 10 is 1. This table does not comprehend that fact. 


7. Suppress #VE. If the “EPT-violation #VE" VM-execution control is 0, this bit is ignored. 
8. Supervisor shadow-stack page. If bit 7 of the EPTP is O, this bit is ignored. 
9. Sub-page write permissions. If the “sub-page write permissions for EPT” VM-execution control is 0, this bit is ignored. 
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The access is a data write and, for any byte to be written, bit 1 (write access) was clear in any of the EPT 
paging-structure entries used to translate the guest-physical address of the byte. Writes by the logical 
processor to guest paging structures to update accessed and dirty flags are considered to be data writes. 


If bit 6 of the EPT pointer (EPTP) is 1 (enabling accessed and dirty flags for EPT), processor accesses to guest 
paging-structure entries are treated as writes with regard to EPT violations. Thus, if bit 1 is clear in any of the 
EPT paging-structure entries used to translate the guest-physical address of a guest paging-structure entry, an 
attempt to use that entry to translate a linear address causes an EPT violation. 


(This does not apply to loads of the PDPTE registers by the MOV to CR instruction for PAE paging; see Section 
4.4.1. Those loads of guest PDPTEs are treated as reads and do not cause EPT violations due to a guest-physical 
address not being writable.) 


If the “sub-page write permissions for EPT” VM-execution control is 1, data writes to a guest-physical address 
that would cause an EPT violation (as indicated above) do not do so in certain situations. If the guest-physical 
address is mapped using a 4-KByte page and bit 61 (sub-page write permissions) of the EPT PTE used to map 
the page is 1, writes to certain 128-byte sub-pages may be allowed. See Section 27.2.4 for details. 


The access is an instruction fetch and the EPT paging structures prevent execute access to any of the bytes 
being fetched. Whether this occurs depends upon the setting of the “mode-based execute control for EPT” VM- 
execution control: 


— Ifthe control is 0, an instruction fetch from a byte is prevented if bit 2 (execute access) was clear in any of 
the EPT paging-structure entries used to translate the guest-physical address of the byte. 


— Ifthe control is 1, an instruction fetch from a byte is prevented in either of the following cases: 


¢ Paging maps the linear address of the byte as a supervisor-mode address and bit 2 (execute access for 
supervisor-mode linear addresses) was clear in any of the EPT paging-structure entries used to 
translate the guest-physical address of the byte. 


Paging maps a linear address as a Supervisor-mode address if the U/S flag (bit 2) is 0 in at least one of 
the paging-structure entries controlling the translation of the linear address. 


¢* Paging maps the linear address of the byte as a user-mode address and bit 10 (execute access for user- 
mode linear addresses) was clear in any of the EPT paging-structure entries used to translate the guest- 
physical address of the byte. 


Paging maps a linear address as a user-mode address if the U/S flag is 1 in all of the paging-structure 
entries controlling the translation of the linear address. If paging is disabled (CRO.PG = 0), every linear 
address is a user-mode address. 


If supervisor shadow-stack control is enabled (by setting bit 7 of EPTP), the access is a supervisor shadow- 
stack access, and the EPT paging-structure entries used to translate the guest-physical address of the access 
disallow supervisor shadow-stack accesses. Such an access is disallowed if any of the following hold: 


— Bit 0 (read access) is clear in any EPT paging-structure entry used to translate the guest-physical address 
of the access. 

— Bit 1 (write access) is clear in any EPT paging-structure entry that references an EPT paging structure in the 
translation of the guest-physical address. (Clearing bit 1 in the EPT paging-structure entry that maps the 
page of the guest-physical address does not disallow shadow-stack reads and writes. ) 

— Bit 60 (supervisor-shadow stack access) is clear in the EPT paging-structure entry that maps the page of 
the guest-physical address. 


Supervisor shadow-stack control and the supervisor-shadow stack access bits in EPT paging-structure entries 
do not affect other accesses (including user shadow-stack accesses). 


27.2.3.3 Prioritization of EPT Misconfigurations and EPT Violations 


The translation of a linear address to a physical address requires one or more translations of guest-physical 
addresses using EPT (see Section 27.2.1). This section specifies the relative priority of EPT-induced VM exits with 
respect to each other and to other events that may be encountered when accessing memory using a linear address. 


For an access to a guest-physical address, determination of whether an EPT misconfiguration or an EPT violation 
occurs is based on an iterative process: 
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An EPT paging-structure entry is read (initially, this is an EPT PML4 entry): 


If the entry is not present (see Section 27.2.2), an EPT violation occurs. 


If the entry is present but its contents are not configured properly (see Section 27.2.3.1), an EPT miscon- 
figuration occurs. 


If the entry is present and its contents are configured properly, operation depends on whether the entry 
references another EPT paging structure (whether it is an EPT PDE with bit 7 set to 1 or an EPT PTE): 


i) Ifthe entry does reference another EPT paging structure, an entry from that structure is accessed; 
step 1 is executed for that other entry. 


ii) Otherwise, the entry is used to produce the ultimate physical address (the translation of the original 
guest-physical address); step 2 is executed. 


Once the ultimate physical address is determined, the privileges determined by the EPT paging-structure 
entries are evaluated: 


a. 


If the access to the guest-physical address is not allowed by these privileges (see Section 27.2.3.2), an EPT 
violation occurs. 


If the access to the guest-physical address is allowed by these privileges, memory is accessed using the 
ultimate physical address. 


If CRO.PG = 1, the translation of a linear address is also an iterative process, with the processor first accessing an 
entry in the guest paging structure referenced by the guest-physical address in CR3 (or, if PAE paging is in use, the 
guest-physical address in the appropriate PDPTE register), then accessing an entry in another guest paging struc- 
ture referenced by the guest-physical address in the first guest paging-structure entry, etc. Each guest-physical 
address is itself translated using EPT and may cause an EPT-induced VM exit. The following items detail how page 
faults and EPT-induced VM exits are recognized during this iterative process: 


1. 


An attempt is made to access a guest paging-structure entry with a guest-physical address (initially, the 
address in CR3 or PDPTE register). 


a. 


If the access fails because of an EPT misconfiguration or an EPT violation (see above), an EPT-induced 
VM exit occurs. 


If the access does not cause an EPT-induced VM exit, bit 0 (the present flag) of the entry is consulted: 
i) Ifthe present flag is O or any reserved bit is set, a page fault occurs. 


ii) If the present flag is 1, no reserved bit is set, operation depends on whether the entry references 
another guest paging structure (whether it is a guest PDE with PS = 1 or a guest PTE): 


¢ If the entry does reference another guest paging structure, an entry from that structure is 
accessed; step 1 is executed for that other entry. 


¢ Otherwise, the entry is used to produce the ultimate guest-physical address (the translation of the 
original linear address); step 2 is executed. 


Once the ultimate guest-physical address is determined, the privileges determined by the guest paging- 
structure entries are evaluated: 


a. 


If the access to the linear address is not allowed by these privileges (e.g., it was a write to a read-only 
page), a page fault occurs. 


If the access to the linear address is allowed by these privileges, an attempt is made to access memory at 
the ultimate guest-physical address: 


i) If the access fails because of an EPT misconfiguration or an EPT violation (see above), an EPT-induced 
VM exit occurs. 


ii) If the access does not cause an EPT-induced VM exit, memory is accessed using the ultimate physical 
address (the translation, using EPT, of the ultimate guest-physical address). 


If CRO.PG = 0, a linear address is treated as a guest-physical address and is translated using EPT (see above). This 
process, if it completes without an EPT violation or EPT misconfiguration, produces a physical address and deter- 


1. 


This is a simplification of the more detailed description given in Section 27.2.2. 
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mines the privileges allowed by the EPT paging-structure entries. If these privileges do not allow the access to the 
physical address (see Section 27.2.3.2), an EPT violation occurs. Otherwise, memory is accessed using the physical 
address. 


27.2.4 Sub-Page Write Permissions 


Section 27.2.3.2 explained how EPT enforces the access rights for guest-physical addresses using EPT violations. 
Since these access rights are determined using the EPT paging-structure entries that are used to translate a guest- 
physical address, their granularity is limited to that which is used to map pages (1-GByte, 2-MByte, or 4-KByte). 


The sub-page write-permission feature allows the control of write accesses to guest-physical addresses to be 
controlled at finer granularity. Sub-page write permissions allow write accesses to be controlled at the granularity 
of naturally aligned 128-byte sub-pages. Specifically, the feature allows writes to selected sub-pages of 4-KByte 
page that would otherwise not be writable. 


Sub-page write permissions are enabled setting the “sub-page write permissions for EPT” VM-execution control to 
1. The remainder of this section describes changes to processor operation with this control setting. 


Section 27.2.4.1 identifies the data accesses that are eligible for sub-page write permissions. Section 27.2.4.2 
explains how the processor determines whether to allow such an access. 


27.2.4.1 Write Accesses That Are Eligible for Sub-Page Write Permissions 


A guest-physical address is eligible for sub-page write permissions if writes to it would be disallowed following 

Section 27.2.3.2: bit 1 (write access) is clear in any of the EPT paging-structure entries used to translate the guest- 
physical address. Guest-physical addresses to which writes would be disallowed for other reasons (e.g., the trans- 
lation encounters an EPT paging-structure entry that is not present) are not eligible for sub-page write permissions. 


In addition, a guest-physical address is eligible for sub-page write permissions only if it is mapped using a 4-KByte 
page and bit 61 (sub-page write permissions) of the EPT PTE used to map the page is 1. (Guest-physical addresses 
mapped with larger pages are not eligible for sub-page write permissions. ) 


For some memory accesses, the processor ignores bit 61 in an EPT PTE used to map a 4-KByte page and does not 
apply sub-page write permissions to the access. (In such a case, the access causes an EPT violation when indicated 
by the conditions given in Section 27.2.3.2.) Sub-page write permissions never apply to the following accesses: 


¢ Awrite access performed within a transactional region. 


e Awrite access by an enclave to an address within the enclave's ELRANGE. (Sub-page write permissions may 
apply to write accesses by an enclaves to addresses outside its ELRANGE.) 


e Awrite access to the enclave page cache (EPC) by an Intel SGX instruction. 
¢ Awrite access to a guest paging structure to update an accessed or dirty flag. 


° Processor accesses to guest paging-structure entries when accessed and dirty flags for EPT are enabled (such 
accesses are treated as writes with regard to EPT violations. 


There are additional accesses to which sub-page write permissions might not be applied (behavior is model- 
specific). The following items enumerate examples: 


¢ Awrite access that crosses two 4-KByte pages. In this case, sub-page permissions may be applied to neither or 
to only one of the pages. (There is no write to either page unless the write is allowed to both pages.) 


¢ Awrite access by an instruction that performs multiple write accesses (Sub-page write permissions are 
intended principally for basic instructions such as AND, MOV, OR, TEST, XCHG, and XOR). 


If a guest-physical address is eligible for sub-page write permissions, the processor determines whether to allow 
write to the address using the process described in Section 27.2.4.2. 


If a guest-physical address is eligible for sub-page write permissions and that address translates to an address on 
the APIC-access page (see Section 28.4), the processor may treat a write access to the address as if the “virtualize 
APIC accesses” VM-execution control were 0. For that reason, it is recommended that software not configure any 
guest-physical address that translates to an address on the APIC-access page to be eligible for sub-page write 
permissions. 
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27.2.4.2 Determining an Access’s Sub-Page Write Permission 


Sub-page write permissions control write accesses individually to each of the 32 128-byte sub-pages of a 4-KByte 
page. Bits 11:7 of guest-physical address identify the sub-page. 


For each guest-physical address eligible for sub-page write permissions, there is a 64-bit sub-page permission 
vector (SPP vector). All addresses on a 4-KByte page use the same SPP vector. If an address’s sub-page number 
(bits 11:7 of the address) is S, writes to address are allowed if and only if bit 2S of the sub-page permission is set 
to 1. (The bits at odd positions in a SPP vector are not used and must be zero.) 


Each page’s SPP vector is located in memory. For a write to a guest-physical address eligible for sub-page write 
permissions, the processor uses the following process to locate the address’s SPP vector: 


1. The SPPTP (sub-page-permission-table pointer) VM-execution control field contains the physical address of the 
4-KByte root SPP table (SSPL4 table). Bits 47:39 of the guest-physical address identify a 64-bit entry in that 
table, called an SPPL4E. 


2. A4-KByte SPPL3 table is located at the physical address in the selected SPPL4E. Bits 38:30 of the guest- 
physical address identify a 64-bit entry in that table, called an SPPL3E. 


3. A4-KByte SPPL2 table is located at the physical address in the selected SPPL3E. Bits 29:21 of the guest- 
physical address identify a 64-bit entry in that table, called an SPPL2E. 


4. A4-KByte SPP-vector table (SSPL1 table) is located at the physical address in the selected SPPL2E. Bits 20:12 
of the guest-physical address identify the 64-bit SPP vector for the address. As noted earlier, bit 2S of the sub- 
page permission vector determines whether the address may be written, where S is the value of address 
bits 11:7. 


(The memory type used to access these tables is reported in bits 53:50 of the IA32_VMX_BASIC MSR. See 
Appendix A.1.) 


A write access to multiple 128-byte sub-pages on a single 4-KByte page is allowed only if the indicated bit in the 
page’s SPP vector for each of those sub-pages. The following items apply to cases in which an access writes to two 
4-KByte pages: 


¢ Ifa write to either page would be disallowed according to Section 28.2.3.2, the access might be disallowed 
even if the guest-physical address of that page is eligible for sub-page write permissions. (This behavior is 
model-specific. ) 


° The access is allowed only if, for each page, either (1) a write to the page would be allowed following Section 
28.2.3.2; or (2) both (a) the guest-physical address of that page is eligible for sub-page write permissions; 
and (b) the page’s sub-page vector allows the write (as described above). 


Bit 0 of each entry (SPPL4E, SPPL3E, or SPPL2E) is the entry’s valid bit. If the process above accesses an entry in 
which this bit is 0, the process stops and the logical processor incurs an SPP miss. 


In each entry (SPPL4E, SPPL4E, or SPPL2E), bits 11:1 are reserved, as are bits 63:N, where N is the processor's 
physical-address width. If the process above accesses an entry in which the valid bit is 1 and in which some 
reserved bit is set, the process stops and the logical processor incurs an SPP misconfiguration. Bits in an SPP 
vector in odd positions are also reserved; an SPP misconfiguration occurs also any of those bits are set in the final 
SPP vector. 


SPP misses and SPP misconfigurations are called SPP-related events and cause VM exits. 


27.2.5 Accessed and Dirty Flags for EPT 


The Intel 64 architecture supports accessed and dirty flags in ordinary paging-structure entries (see Section 
4.8). Some processors also support corresponding flags in EPT paging-structure entries. Software should read the 
VMX capability MSR IA32_VMX_EPT_VPID_CAP (see Appendix A.10) to determine whether the processor supports 
this feature. 


Software can enable accessed and dirty flags for EPT using bit 6 of the extended-page-table pointer (EPTP), a VM- 
execution control field (see Table 23-9 in Section 23.6.11). If this bit is 1, the processor will set the accessed and 
dirty flags for EPT as described below. In addition, setting this flag causes processor accesses to guest paging- 
structure entries to be treated as writes (see below and Section 27.2.3.2). 
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For any EPT paging-structure entry that is used during guest-physical-address translation, bit 8 is the accessed 
flag. For a EPT paging-structure entry that maps a page (as opposed to referencing another EPT paging structure), 
bit 9 is the dirty flag. 


Whenever the processor uses an EPT paging-structure entry as part of guest-physical-address translation, it sets 
the accessed flag in that entry (if it is not already set). 


Whenever there is a write to a guest-physical address, the processor sets the dirty flag (if it is not already set) in 
the EPT paging-structure entry that identifies the final physical address for the guest-physical address (either an 
EPT PTE or an EPT paging-structure entry in which bit 7 is 1). 


When accessed and dirty flags for EPT are enabled, processor accesses to guest paging-structure entries are 
treated as writes (see Section 27.2.3.2). Thus, such an access will cause the processor to set the dirty flag in the 
EPT paging-structure entry that identifies the final physical address of the guest paging-structure entry. 


(This does not apply to loads of the PDPTE registers for PAE paging by the MOV to CR instruction; see Section 4.4.1. 
Those loads of guest PDPTEs are treated as reads and do not cause the processor to set the dirty flag in any EPT 
paging-structure entry.) 


These flags are “sticky,” meaning that, once set, the processor does not clear them; only software can clear them. 


A processor may cache information from the EPT paging-structure entries in TLBs and paging-structure caches (see 
Section 27.3). This fact implies that, if software changes an accessed flag or a dirty flag from 1 to 0, the processor 
might not set the corresponding bit in memory on a subsequent access using an affected guest-physical address. 


27.2.6 Page-Modification Logging 


When accessed and dirty flags for EPT are enabled, software can track writes to guest-physical addresses using a 
feature called page-modification logging. 


Software can enable page-modification logging by setting the “enable PML” VM-execution control (see Table 23-7 
in Section 23.6.2). When this control is 1, the processor adds entries to the page-modification log as described 
below. The page-modification log is a 4-KByte region of memory located at the physical address in the PML address 
VM-execution control field. The page-modification log consists of 512 64-bit entries; the PML index VM-execution 
control field indicates the next entry to use. 


Before allowing a guest-physical access, the processor may determine that it first needs to set an accessed or dirty 
flag for EPT (see Section 27.2.5). When this happens, the processor examines the PML index. If the PML index is 
not in the range 0-511, there is a page-modification log-full event and a VM exit occurs. In this case, the 
accessed or dirty flag is not set, and the guest-physical access that triggered the event does not occur. 


If instead the PML index is in the range 0-511, the processor proceeds to update accessed or dirty flags for EPT as 
described in Section 27.2.5. If the processor updated a dirty flag for EPT (changing it from 0 to 1), it then operates 
as follows: 


1. The guest-physical address of the access is written to the page-modification log. Specifically, the guest-physical 
address is written to physical address determined by adding 8 times the PML index to the PML address. 
Bits 11:0 of the value written are always 0 (the guest-physical address written is thus 4-KByte aligned). 


2. The PML index is decremented by 1 (this may cause the value to transition from 0 to FFFFH). 


Because the processor decrements the PML index with each log entry, the value may transition from 0 to FFFFH. At 
that point, no further logging will occur, as the processor will determine that the PML index is not in the range 0- 
511 and will generate a page-modification log-full event (see above). 


27.2.7 EPT and Memory Typing 


This section specifies how a logical processor determines the memory type use for a memory access while EPT is in 
use. (See Chapter 11, “Memory Cache Control” of Inte/® 64 and IA-32 Architectures Software Developer’s Manual, 
Volume 3A for details of memory typing in the Intel 64 architecture.) Section 27.2.7.1 explains how the memory 

type is determined for accesses to the EPT paging structures. Section 27.2.7.2 explains how the memory type is 

determined for an access using a guest-physical address that is translated using EPT. 
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27.2.7.1| Memory Type Used for Accessing EPT Paging Structures 


This section explains how the memory type is determined for accesses to the EPT paging structures. The determi- 
nation is based first on the value of bit 30 (cache disable—CD) in control register CRO: 


¢ If CRO.CD = 0, the memory type used for any such reference is the EPT paging-structure memory type, which 
is specified in bits 2:0 of the extended-page-table pointer (EPTP), a VM-execution control field (see Section 
23.6.11). A value of 0 indicates the uncacheable type (UC), while a value of 6 indicates the write-back type 
(WB). Other values are reserved. 


¢ If CRO.CD = 1, the memory type used for any such reference is uncacheable (UC). 
The MTRRs have no effect on the memory type used for an access to an EPT paging structure. 


27.2.7.2 Memory Type Used for Translated Guest-Physical Addresses 


The effective memory type of a memory access using a guest-physical address (an access that is translated 
using EPT) is the memory type that is used to access memory. The effective memory type is based on the value of 
bit 30 (cache disable—CD) in control register CRO; the last EPT paging-structure entry used to translate the guest- 
physical address (either an EPT PDE with bit 7 set to 1 or an EPT PTE); and the PAT memory type (see below): 


¢ The PAT memory type depends on the value of CRO.PG: 
— If CRO.PG = 0, the PAT memory type is WB (writeback).4 


— If CRO.PG = 1, the PAT memory type is the memory type selected from the IA32_PAT MSR as specified in 
Section 11.12.3, “Selecting a Memory Type from the PAT”.? 


¢ The EPT memory type is specified in bits 5:3 of the last EPT paging-structure entry: 0 = UC; 1 = WC; 4 = 
WT; 5 = WP; and 6 = WB. Other values are reserved and cause EPT misconfigurations (see Section 27.2.3). 


° If CRO.CD = 0, the effective memory type depends upon the value of bit 6 of the last EPT paging-structure 
entry: 


— If the value is 0, the effective memory type is the combination of the EPT memory type and the PAT 
memory type specified in Table 11-7 in Section 11.5.2.2, using the EPT memory type in place of the MTRR 
memory type. 


— Ifthe value is 1, the memory type used for the access is the EPT memory type. The PAT memory type is 
ignored. 


¢ If CRO.CD = 1, the effective memory type is UC. 
The MTRRs have no effect on the memory type used for an access to a guest-physical address. 


27.3. CACHING TRANSLATION INFORMATION 


Processors supporting Intel® 64 and IA-32 architectures may accelerate the address-translation process by 
caching on the processor data from the structures in memory that control that process. Such caching is discussed 
in Section 4.10, “Caching Translation Information” in the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 3A. The current section describes how this caching interacts with the VMX architecture. 


The VPID and EPT features of the architecture for VMX operation augment this caching architecture. EPT defines 
the guest-physical address space and defines translations to that address space (from the linear-address space) 


1. If the capability MSR 1A32_VMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, CRO.PG can be 0 in VMX non-root 
operation only if the “unrestricted guest” VM-execution control and bit 31 of the primary processor-based VM-execution controls 
are both 1. 


2. Table 11-11 in Section 11.12.3, “Selecting a Memory Type from the PAT” illustrates how the PAT memory type is selected based on 
the values of the PAT, PCD, and PWT bits in a page-table entry (or page-directory entry with PS = 1). For accesses to a guest paging- 
structure entry X, the PAT memory type is selected from the table by using a value of O for the PAT bit with the values of PCD and 
PWT from the paging-structure entry Y that references X (or from CR3 if X is in the root paging structure). With PAE paging, the PAT 
memory type for accesses to the PDPTEs is WB. 


Vol. 3C 27-17 


VMX SUPPORT FOR ADDRESS TRANSLATION 


and from that address space (to the physical-address space). Both features control the ways in which a logical 
processor may create and use information cached from the paging structures. 


Section 27.3.1 describes the different kinds of information that may be cached. Section 27.3.2 specifies when such 
information may be cached and how it may be used. Section 27.3.3 details how software can invalidate cached 
information. 


27.3.1 Information That May Be Cached 


Section 4.10, “Caching Translation Information” in Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 3A identifies two kinds of translation-related information that may be cached by a logical 
processor: translations, which are mappings from linear page numbers to physical page frames, and paging- 
structure caches, which map the upper bits of a linear page number to information from the paging-structure 
entries used to translate linear addresses matching those upper bits. 


The same kinds of information may be cached when VPIDs and EPT are in use. A logical processor may cache and 
use such information based on its function. Information with different functionality is identified as follows: 


* Linear mappings. There are two kinds: 


— Linear translations. Each of these is a mapping from a linear page number to the physical page frame to 
which it translates, along with information about access privileges and memory typing. 


— Linear paging-structure-cache entries. Each of these is a mapping from the upper portion of a linear 
address to the physical address of the paging structure used to translate the corresponding region of the 
linear-address space, along with information about access privileges. For example, bits 47:39 of a linear 
address would map to the address of the relevant page-directory-pointer table. 


Linear mappings do not contain information from any EPT paging structure. 
* Guest-physical mappings.2 There are two kinds: 


— Guest-physical translations. Each of these is a mapping from a guest-physical page number to the physical 
page frame to which it translates, along with information about access privileges and memory typing. 


— Guest-physical paging-structure-cache entries. Each of these is a mapping from the upper portion of a 
guest-physical address to the physical address of the EPT paging structure used to translate the corre- 
sponding region of the guest-physical address space, along with information about access privileges. 


The information in guest-physical mappings about access privileges and memory typing is derived from EPT 
paging structures. 


* Combined mappings.? There are two kinds: 


— Combined translations. Each of these is a mapping from a linear page number to the physical page frame to 
which it translates, along with information about access privileges and memory typing. 


— Combined paging-structure-cache entries. Each of these is a mapping from the upper portion of a linear 
address to the physical address of the paging structure used to translate the corresponding region of the 
linear-address space, along with information about access privileges. 


The information in combined mappings about access privileges and memory typing is derived from both guest 
paging structures and EPT paging structures. 


Guest-physical mappings and combined mappings may also include SPP vectors and information about the data 
structures used to locate SPP vectors (see Section 27.2.4.2). 


27.3.2 Creating and Using Cached Translation Information 


The following items detail the creation of the mappings described in the previous section:* 


1. Earlier versions of this manual used the term “VPID-tagged” to identify linear mappings. 
2. Earlier versions of this manual used the term “EPTP-tagged” to identify quest-physical mappings. 
3. Earlier versions of this manual used the term “dual-tagged” to identify combined mappings. 
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The following items describe the creation of mappings while EPT is not in use (including execution outside VMX 
non-root operation): 


— Linear mappings may be created. They are derived from the paging structures referenced (directly or 
indirectly) by the current value of CR3 and are associated with the current VPID and the current PCID. 


— Nolinear mappings are created with information derived from paging-structure entries that are not present 
(bit 0 is 0) or that set reserved bits. For example, if a PTE is not present, no linear mapping are created for 
any linear page number whose translation would use that PTE. 


— No guest-physical or combined mappings are created while EPT is not in use. 
The following items describe the creation of mappings while EPT is in use: 


— Guest-physical mappings may be created. They are derived from the EPT paging structures referenced 
(directly or indirectly) by bits 51:12 of the current EPTP. These 40 bits contain the address of the EPT-PML4- 
table. (the notation EP4TA refers to those 40 bits). Newly created guest-physical mappings are associated 
with the current EP4TA. 


— Combined mappings may be created. They are derived from the EPT paging structures referenced (directly 
or indirectly) by the current EP4TA. If CRO.PG = 1, they are also derived from the paging structures 
referenced (directly or indirectly) by the current value of CR3. They are associated with the current VPID, 
the current PCID, and the current EP4TA.! No combined paging-structure-cache entries are created if 
CRO.PG = 0. 


— No guest-physical mappings or combined mappings are created with information derived from EPT paging- 
structure entries that are not present (see Section 27.2.2) or that are misconfigured (see Section 
27.2.3.1). 


— No combined mappings are created with information derived from guest paging-structure entries that are 
not present or that set reserved bits. 


— No linear mappings are created while EPT is in use. 


The following items detail the use of the various mappings: 


If EPT is not in use (e.g., when outside VMX non-root operation), a logical processor may use cached mappings 
as follows: 


— For accesses using linear addresses, it may use linear mappings associated with the current VPID and the 
current PCID. It may also use global TLB entries (linear mappings) associated with the current VPID and 
any PCID. 


— No guest-physical or combined mappings are used while EPT is not in use. 
If EPT is in use, a logical processor may use cached mappings as follows: 


— For accesses using linear addresses, it may use combined mappings associated with the current VPID, the 
current PCID, and the current EP4TA. It may also use global TLB entries (combined mappings) associated 
with the current VPID, the current EP4TA, and any PCID. 


— For accesses using guest-physical addresses, it may use guest-physical mappings associated with the 
current EP4TA. 


— No linear mappings are used while EPT is in use. 


This section associated cached information with the current VPID and PCID. If PCIDs are not supported or are not being used (e.g., 
because CR4.PCIDE = 0), all the information is implicitly associated with PCID OOOH; see Section 4.10.1, “Process-Context Identifiers 
(PCIDs),” in Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 3A. 


. At any given time, a logical processor may be caching combined mappings for a VPID and a PCID that are associated with different 


EPATAs. Similarly, it may be caching combined mappings for an EP4TA that are associated with different VPIDs and PCIDs. 


If the capability MSR IA32_WMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, CRO.PG can be O in VMX non-root 
operation only if the “unrestricted quest” VM-execution control and bit 31 of the primary processor-based VM-execution controls 
are both 1. 
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27.3.3.‘ Invalidating Cached Translation Information 


Software modifications of paging structures (including EPT paging structures and the data structures used to locate 
SPP vectors) may result in inconsistencies between those structures and the mappings cached by a logical 
processor. Certain operations invalidate information cached by a logical processor and can be used to eliminate 
such inconsistencies. 


27.3.3.1 Operations that Invalidate Cached Mappings 


The following operations invalidate cached mappings as indicated: 


Operations that architecturally invalidate entries in the TLBs or paging-structure caches independent of VMX 
operation (e.g., the INVLPG and INVPCID instructions) invalidate linear mappings and combined mappings. 
They are required to do so only for the current VPID (but, for combined mappings, all EP4TAs). Linear mappings 
for the current VPID are invalidated even if EPT is in use.* Combined mappings for the current VPID are 
invalidated even if EPT is not in use.? 


An EPT violation invalidates any guest-physical mappings (associated with the current EP4TA) that would be 
used to translate the guest-physical address that caused the EPT violation. If that guest-physical address was 
the translation of a linear address, the EPT violation also invalidates any combined mappings for that linear 
address associated with the current PCID, the current VPID and the current EP4TA. 


If the “enable VPID” VM-execution control is 0, VM entries and VM exits invalidate linear mappings and 
combined mappings associated with VPID OOOOH (for all PCIDs). Combined mappings for VPID OOOOH are 
invalidated for all EP4TAs. 


Execution of the INVVPID instruction invalidates linear mappings and combined mappings. Invalidation is based 
on instruction operands, called the INVVPID type and the INVVPID descriptor. Four INVVPID types are currently 
defined: 


— Individual-address. If the INVVPID type is 0, the logical processor invalidates linear mappings and 
combined mappings associated with the VPID specified in the INVVPID descriptor and that would be used 
to translate the linear address specified in of the INVVPID descriptor. Linear mappings and combined 
mappings for that VPID and linear address are invalidated for all PCIDs and, for combined mappings, all 
EP4TAs. (The instruction may also invalidate mappings associated with other VPIDs and for other linear 
addresses.) 


— Single-context. If the INVVPID type is 1, the logical processor invalidates all linear mappings and 
combined mappings associated with the VPID specified in the INVVPID descriptor. Linear mappings and 
combined mappings for that VPID are invalidated for all PCIDs and, for combined mappings, all EP4TAs. 
(The instruction may also invalidate mappings associated with other VPIDs.) 


— All-context. If the INVVPID type is 2, the logical processor invalidates linear mappings and combined 
mappings associated with all VPIDs except VPID O000H and with all PCIDs. (The instruction may also 
invalidate linear mappings with VPID OOOOH.) Combined mappings are invalidated for all EP4TAs. 


— Single-context-retaining-globals. If the INVVPID type is 3, the logical processor invalidates linear 
mappings and combined mappings associated with the VPID specified in the INVVPID descriptor. Linear 
mappings and combined mappings for that VPID are invalidated for all PCIDs and, for combined mappings, 
all EP4TAs. The logical processor is not required to invalidate information that was used for global transla- 
tions (although it may do so). See Section 4.10, “Caching Translation Information” for details regarding 
global translations. (The instruction may also invalidate mappings associated with other VPIDs.) 


See Chapter 29 for details of the INVVPID instruction. See Section 27.3.3.3 for guidelines regarding use of this 
instruction. 


See Section 4.10.4, “Invalidation of TLBs and Paging-Structure Caches,” in the Intel® 64 and IA-32 Architectures Software Devel- 
oper’s Manual, Volume 3A for an enumeration of operations that architecturally invalidate entries in the TLBs and paging-structure 
caches independent of VMX operation. 


While no linear mappings are created while EPT is in use, a logical processor may retain, while EPT is in use, linear mappings (for the 
same VPID as the current one) there were created earlier, when EPT was not in use. 

While no combined mappings are created while EPT is not in use, a logical processor may retain, while EPT is in not use, combined 
mappings (for the same VPID as the current one) there were created earlier, when EPT was in use. 
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¢ ~=Execution of the INVEPT instruction invalidates guest-physical mappings and combined mappings. Invalidation 
is based on instruction operands, called the INVEPT type and the INVEPT descriptor. Two INVEPT types are 
currently defined: 


— Single-context. If the INVEPT type is 1, the logical processor invalidates all guest-physical mappings and 
combined mappings associated with the EP4TA specified in the INVEPT descriptor. Combined mappings for 
that EP4TA are invalidated for all VPIDs and all PCIDs. (The instruction may invalidate mappings associated 
with other EP4TAs. ) 


— All-context. If the INVEPT type is 2, the logical processor invalidates guest-physical mappings and 
combined mappings associated with all EP4TAs (and, for combined mappings, for all VPIDs and PCIDs). 


See Chapter 29 for details of the INVEPT instruction. See Section 27.3.3.4 for guidelines regarding use of this 
instruction. 


¢ A power-up or a reset invalidates all linear mappings, guest-physical mappings, and combined mappings. 


27.3.3.2 Operations that Need Not Invalidate Cached Mappings 


The following items detail cases of operations that are not required to invalidate certain cached mappings: 


* Operations that architecturally invalidate entries in the TLBs or paging-structure caches independent of VMX 
operation are not required to invalidate any guest-physical mappings. 


¢ The INVVPID instruction is not required to invalidate any guest-physical mappings. 
¢ The INVEPT instruction is not required to invalidate any linear mappings. 


¢ VMxX transitions are not required to invalidate any guest-physical mappings. If the “enable VPID” VM-execution 
control is 1, VMX transitions are not required to invalidate any linear mappings or combined mappings. 


¢ The VMXOFF and VMXON instructions are not required to invalidate any linear mappings, guest-physical 
mappings, or combined mappings. 


A logical processor may invalidate any cached mappings at any time. For this reason, the operations identified 
above may invalidate the indicated mappings despite the fact that doing so is not required. 


27.3.3.3 Guidelines for Use of the INVVPID Instruction 
The need for VMM software to use the INVVPID instruction depends on how that software is virtualizing memory. 


If EPT is not in use, it is likely that the VMM is virtualizing the guest paging structures. Such a VMM may configure 
the VMCS so that all or some of the operations that invalidate entries the TLBs and the paging-structure caches 
(e.g., the INVLPG instruction) cause VM exits. If VMM software is emulating these operations, it may be necessary 
to use the INVVPID instruction to ensure that the logical processor’s TLBs and the paging-structure caches are 
appropriately invalidated. 


Requirements of when software should use the INVVPID instruction depend on the specific algorithm being used for 
page-table virtualization. The following items provide guidelines for software developers: 


¢ Emulation of the INVLPG instruction may require execution of the INVVPID instruction as follows: 
— The INVVPID type is individual-address (0). 


— The VPID in the INVVPID descriptor is the one assigned to the virtual processor whose execution is being 
emulated. 


— The linear address in the INVVPID descriptor is that of the operand of the INVLPG instruction being 
emulated. 


* Some instructions invalidate all entries in the TLBs and paging-structure caches—except for global translations. 
An example is the MOV to CR3 instruction. (See Section 4.10, “Caching Translation Information” in the Inte/® 
64 and IA-32 Architectures Software Developer’s Manual, Volume 3A for details regarding global translations. ) 
Emulation of such an instruction may require execution of the INVVPID instruction as follows: 


— The INVVPID type is single-context-retaining-globals (3). 


— The VPID in the INVVPID descriptor is the one assigned to the virtual processor whose execution is being 
emulated. 
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* Some instructions invalidate all entries in the TLBs and paging-structure caches—including for global transla- 
tions. An example is the MOV to CR4 instruction if the value of value of bit 4 (page global enable—PGE) is 
changing. Emulation of such an instruction may require execution of the INVVPID instruction as follows: 


— The INVVPID type is single-context (1). 


— The VPID in the INVVPID descriptor is the one assigned to the virtual processor whose execution is being 
emulated. 


If EPT is not in use, the logical processor associates all mappings it creates with the current VPID, and it will use 
such mappings to translate linear addresses. For that reason, a VMM should not use the same VPID for different 
non-EPT guests that use different page tables. Doing so may result in one guest using translations that pertain to 
the other. 


If EPT is in use, the instructions enumerated above might not be configured to cause VM exits and the VMM might 
not be emulating them. In that case, executions of the instructions by guest software properly invalidate the 
required entries in the TLBs and paging-structure caches (see Section 27.3.3.1); execution of the INVVPID instruc- 
tion is not required. 


If EPT is in use, the logical processor associates all mappings it creates with the value of bits 51:12 of current EPTP. 
If a VMM uses different EPTP values for different guests, it may use the same VPID for those guests. Doing so 
cannot result in one guest using translations that pertain to the other. 


The following guidelines apply more generally and are appropriate even if EPT is in use: 


° As detailed in Section 28.4.5, an access to the APIC-access page might not cause an APIC-access VM exit if 
software does not properly invalidate information that may be cached from the paging structures. If, at one 
time, the current VPID on a logical processor was a non-zero value X, it is recommended that software use the 
INVVPID instruction with the “single-context” INVVPID type and with VPID X in the INVVPID descriptor before 
a VM entry on the same logical processor that establishes VPID X and either (a) the “virtualize APIC accesses” 
VM-execution control was changed from 0 to 1; or (b) the value of the APIC-access address was changed. 


¢* Software can use the INVVPID instruction with the “all-context” INVVPID type immediately after execution of 
the VMXON instruction or immediately prior to execution of the VMXOFF instruction. Either prevents potentially 
undesired retention of information cached from paging structures between separate uses of VMX operation. 


27.3.3.4 Guidelines for Use of the INVEPT Instruction 


The following items provide guidelines for use of the INVEPT instruction to invalidate information cached from the 
EPT paging structures. 


* Software should use the INVEPT instruction with the “single-context” INVEPT type after making any of the 
following changes to an EPT paging-structure entry (the INVEPT descriptor should contain an EPTP value that 
references — directly or indirectly — the modified EPT paging structure): 


— Changing any of the privilege bits 2:0 from 1 to 0.! 

— Changing the physical address in bits 51:12. 

— Clearing bit 8 (the accessed flag) if accessed and dirty flags for EPT will be enabled. 

— Foran EPT PDPTE or an EPT PDE, changing bit 7 (which determines whether the entry maps a page). 


— For the last EPT paging-structure entry used to translate a guest-physical address (an EPT PDPTE with bit 7 
set to 1, an EPT PDE with bit 7 set to 1, or an EPT PTE), changing either bits 5:3 or bit 6. (These bits 
determine the effective memory type of accesses using that EPT paging-structure entry; see Section 
27.2.7.) 


— For the last EPT paging-structure entry used to translate a guest-physical address (an EPT PDPTE with bit 7 
set to 1, an EPT PDE with bit 7 set to 1, or an EPT PTE), clearing bit 9 (the dirty flag) if accessed and dirty 
flags for EPT will be enabled. 


¢ Software should use the INVEPT instruction with the “single-context” INVEPT type before a VM entry with an 
EPTP value X such that X[6] = 1 (accessed and dirty flags for EPT are enabled) if the logical processor had 


1. If the “mode-based execute control for EPT” VM-execution control is 1, software should use the INVEPT instruction after changing 
privilege bit 10 from 1 to O. 
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earlier been in VMX non-root operation with an EPTP value Y such that Y[6] = O (accessed and dirty flags for 
EPT are not enabled) and Y[51:12] = X[51:12]. 


Software may use the INVEPT instruction after modifying a present EPT paging-structure entry (see Section 
27.2.2) to change any of the privilege bits 2:0 from 0 to 1.! Failure to do so may cause an EPT violation that 
would not otherwise occur. Because an EPT violation invalidates any mappings that would be used by the 
access that caused the EPT violation (see Section 27.3.3.1), an EPT violation will not recur if the original access 
is performed again, even if the INVEPT instruction is not executed. 


Because a logical processor does not cache any information derived from EPT paging-structure entries that are 
not present (see Section 27.2.2) or misconfigured (see Section 27.2.3.1), itis not necessary to execute INVEPT 
following modification of an EPT paging-structure entry that had been not present or misconfigured. 


As detailed in Section 28.4.5, an access to the APIC-access page might not cause an APIC-access VM exit if 
software does not properly invalidate information that may be cached from the EPT paging structures. If EPT 
was in use on a logical processor at one time with EPTP X, it is recommended that software use the INVEPT 
instruction with the “single-context” INVEPT type and with EPTP X in the INVEPT descriptor before a VM entry 
on the same logical processor that enables EPT with EPTP X and either (a) the “virtualize APIC accesses” VM- 
execution control was changed from 0 to 1; or (b) the value of the APIC-access address was changed. 


Software can use the INVEPT instruction with the “all-context” INVEPT type immediately after execution of the 
VMXON instruction or immediately prior to execution of the VMXOFF instruction. Either prevents potentially 
undesired retention of information cached from EPT paging structures between separate uses of VMX 
operation. 


In a system containing more than one logical processor, software must account for the fact that information from 
an EPT paging-structure entry may be cached on logical processors other than the one that modifies that entry. The 
process of propagating the changes to a paging-structure entry is commonly referred to as “TLB shootdown.” A 
discussion of TLB shootdown appears in Section 4.10.5, “Propagation of Paging-Structure Changes to Multiple 
Processors,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A. 


1. 


If the “mode-based execute control for EPT” VM-execution control is 1, software may use the INVEPT instruction after modifying a 
present EPT paging-structure entry to change privilege bit 10 from O to 1. 
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CHAPTER 28 
APIC VIRTUALIZATION AND VIRTUAL INTERRUPTS 


The VMCS includes controls that enable the virtualization of interrupts and the Advanced Programmable Interrupt 
Controller (APIC). 


When these controls are used, the processor will emulate many accesses to the APIC, track the state of the virtual 
APIC, and deliver virtual interrupts — all in VMX non-root operation with out a VM exit. 


The processor tracks the state of the virtual APIC using a virtual-APIC page identified by the virtual-machine 
monitor (VMM). Section 28.1 discusses the virtual-APIC page and how the processor uses it to track the state of the 
virtual APIC. 


The following are the VM-execution controls relevant to APIC virtualization and virtual interrupts (see Section 23.6 
for information about the locations of these controls): 


* Virtual-interrupt delivery. This controls enables the evaluation and delivery of pending virtual interrupts 
(Section 28.2). It also enables the emulation of writes (memory-mapped or MSR-based, as enabled) to the 
APIC registers that control interrupt prioritization. 


¢ Use TPR shadow. This control enables emulation of accesses to the APIC’s task-priority register (TPR) via CR8 
(Section 28.3) and, if enabled, via the memory-mapped or MSR-based interfaces. 


°® Virtualize APIC accesses. This control enables virtualization of memory-mapped accesses to the APIC 
(Section 28.4) by causing VM exits on accesses to a VMM-specified APIC-access page. Some of the other 
controls, if set, may cause some of these accesses to be emulated rather than causing VM exits. 


° Virtualize x2APIC mode. This control enables virtualization of MSR-based accesses to the APIC (Section 
28.5). 


¢ APIC-register virtualization. This control allows memory-mapped and MSR-based reads of most APIC 
registers (as enabled) by satisfying them from the virtual-APIC page. It directs memory-mapped writes to the 
APIC-access page to the virtual-APIC page, following them by VM exits for VMM emulation. 


* Process posted interrupts. This control allows software to post virtual interrupts in a data structure and send 
a notification to another logical processor; upon receipt of the notification, the target processor will process the 
posted interrupts by copying them into the virtual-APIC page (Section 28.6). 


a a 


“Virtualize APIC accesses”, “virtualize x2APIC mode”, “virtual-interrupt delivery”, and “APIC-register virtualization” 
are all secondary processor-based VM-execution controls. If bit 31 of the primary processor-based VM-execution 
controls is 0, the processor operates as if these controls were all 0. See Section 23.6.2. 


28.1 VIRTUAL APIC STATE 


The virtual-APIC page is a 4-KByte region of memory that the processor uses to virtualize certain accesses to 
APIC registers and to manage virtual interrupts. The physical address of the virtual-APIC page is the virtual-APIC 
address, a 64-bit VM-execution control field in the VMCS (see Section 23.6.8). 


Depending on the settings of certain VM-execution controls, the processor may virtualize certain fields on the 
virtual-APIC page with functionality analogous to that performed by the local APIC. Section 28.1.1 identifies and 
defines these fields. Section 28.1.2, Section 28.1.3, Section 28.1.4, and Section 28.1.5 detail the actions taken to 
virtualize updates to some of these fields. 


With the exception of fields corresponding to virtualized APIC registers (defined in Section 28.1.1), software may 
modify the virtual-APIC page referenced by the current VMCS of a logical processor in VMX non-root operation. 
(This is an exception to the general requirement given in Section 23.11.4.) 


1. In most cases, it is not necessary for a virtual-machine monitor (VMM) to inject virtual interrupts as part of VM entry. 
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28.1.1. ‘Virtualized APIC Registers 


Depending on the setting of certain VM-execution controls, a logical processor may virtualize certain accesses to 
APIC registers using the following fields on the virtual-APIC page: 


°* Virtual task-priority register (VTPR): the 32-bit field located at offset O80H on the virtual-APIC page. 


* Virtual processor-priority register (VPPR): the 32-bit field located at offset OAOH on the virtual-APIC 
page. 

* Virtual end-of-interrupt register (VEOI): the 32-bit field located at offset OBOH on the virtual-APIC page. 

* Virtual interrupt-service register (VISR): the 256-bit value comprising eight non-contiguous 32-bit fields 
at offsets 100H, 110H, 120H, 130H, 140H, 150H, 160H, and 170H on the virtual-APIC page. Bit x of the VISR 
is at bit position (x & 1FH) at offset (100H | ((x & EOH) » 1)). The processor uses only the low 4 bytes of each 
of the 16-byte fields at offsets 100H, 110H, 120H, 130H, 140H, 150H, 160H, and 170H. 

* Virtual interrupt-request register (VIRR): the 256-bit value comprising eight non-contiguous 32-bit fields 
at offsets 200H, 210H, 220H, 230H, 240H, 250H, 260H, and 270H on the virtual-APIC page. Bit x of the VIRR 
is at bit position (x & 1FH) at offset (200H | ((x & EOH) » 1)). The processor uses only the low 4 bytes of each 
of the 16-Byte fields at offsets 200H, 210H, 220H, 230H, 240H, 250H, 260H, and 270H. 

°* Virtual interrupt-command register (VICR_LO): the 32-bit field located at offset 300H on the virtual-APIC 
page 

* Virtual interrupt-command register (VICR_HI): the 32-bit field located at offset 310H on the virtual-APIC 
page. 

The VTPR field virtualizes the TPR whenever the “use TPR shadow” VM-execution control is 1. The other fields indi- 

cated above virtualize the corresponding APIC registers whenever the “virtual-interrupt delivery” VM-execution 

control is 1. 


28.1.2 TPR Virtualization 


The processor performs TPR virtualization in response to the following operations: (1) virtualization of the MOV 
to CR8 instruction; (2) virtualization of a write to offset O80H on the APIC-access page; and (3) virtualization of the 
WRMSR instruction with ECX = 808H. See Section 28.3, Section 28.4.3, and Section 28.5 for details of when TPR 
virtualization is performed. 


The following pseudocode details the behavior of TPR virtualization: 
IF “virtual-interrupt delivery” is 0 
THEN 
IF VTPR[7:4] < TPR threshold (see Section 23.6.8) 
THEN cause VM exit due to TPR below threshold; 
Fl; 
ELSE 
perform PPR virtualization (see Section 28.1.3); 
evaluate pending virtual interrupts (see Section 28.2.1); 
Fl; 


Any VM exit caused by TPR virtualization is trap-like: the instruction causing TPR virtualization completes before 
the VM exit occurs (for example, the value of CS:RIP saved in the guest-state area of the VMCS references the next 
instruction). 


28.1.3. PPR Virtualization 


The processor performs PPR virtualization in response to the following operations: (1) VM entry; (2) TPR virtu- 
alization; and (3) EOI virtualization. See Section 25.3.2.5, Section 28.1.2, and Section 28.1.4 for details of when 
PPR virtualization is performed. 


PPR virtualization uses the guest interrupt status (specifically, SVI; see Section 23.4.2) and VTPR. The following 
pseudocode details the behavior of PPR virtualization: 


IF VTPRI7:4] = SVI[7:4] 
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THEN VPPR := VTPR & FFH; 
ELSE VPPR := SVI & FOH; 
Fl; 


PPR virtualization always clears bytes 3:1 of VPPR. 


PPR virtualization is caused only by TPR virtualization, EOI virtualization, and VM entry. Delivery of a virtual inter- 
rupt also modifies VPPR, but in a different way (see Section 28.2.2). No other operations modify VPPR, even if they 
modify SVI, VISR, or VTPR. 


28.1.4 EOI Virtualization 


The processor performs EOI virtualization in response to the following operations: (1) virtualization of a write to 
offset OBOH on the APIC-access page; and (2) virtualization of the WRMSR instruction with ECX = 80BH. See 
Section 28.4.3 and Section 28.5 for details of when EOI virtualization is performed. EOI virtualization occurs only 
if the “virtual-interrupt delivery” VM-execution control is 1. 


EOI virtualization uses and updates the guest interrupt status (specifically, SVI; see Section 23.4.2). The following 
pseudocode details the behavior of EOI virtualization: 
Vector := SVI; 
VISR[Vector] := 0; (see Section 28.1.1 for definition of VISR) 
IF any bits set in VISR 
THEN SVI := highest index of bit set in VISR 
ELSE SVI := 0; 
Fl; 
perform PPR virtualiation (see Section 28.1.3); 
IF EOl_exit_bitmap[Vector] = 1 (see Section 23.6.8 for definition of EOl_exit_bitmap) 
THEN cause EOl-induced VM exit with Vector as exit qualification; 
ELSE evaluate pending virtual interrupts; (see Section 28.2.1) 
Fl; 
Any VM exit caused by EOI virtualization is trap-like: the instruction causing EOI virtualization completes before 
the VM exit occurs (for example, the value of CS:RIP saved in the guest-state area of the VMCS references the next 
instruction). 


28.1.5  Self-IPI Virtualization 


The processor performs self-IPI virtualization in response to the following operations: (1) virtualization of a 
write to offset 300H on the APIC-access page; and (2) virtualization of the WRMSR instruction with ECX = 83FH. 
See Section 28.4.3 and Section 28.5 for details of when self-IPI virtualization is performed. Self-IPI virtualization 
occurs only if the “virtual-interrupt delivery” VM-execution control is 1. 


Each operation that leads to self-IPI virtualization provides an 8-bit vector (see Section 28.4.3 and Section 28.5). 
Self-IPI virtualization updates the guest interrupt status (specifically, RVI; see Section 23.4.2). The following 
pseudocode details the behavior of self-IPI virtualization: 

VIRR[Vector] := 1; (see Section 28.1.1 for definition of VIRR) 

RVI := max{RVI,Vector}; 

evaluate pending virtual interrupts; (see Section 28.2.1) 


28.2 EVALUATION AND DELIVERY OF VIRTUAL INTERRUPTS 


If the “virtual-interrupt delivery” VM-execution control is 1, certain actions in VMX non-root operation or during 
VM entry cause the processor to evaluate and deliver virtual interrupts. 


Evaluation of virtual interrupts is triggered by certain actions change the state of the virtual-APIC page and is 
described in Section 28.2.1. This evaluation may result in recognition of a virtual interrupt. Once a virtual interrupt 
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is recognized, the processor may deliver it within VMX non-root operation without a VM exit. Virtual-interrupt 
delivery is described in Section 28.2.2. 


28.2.1 Evaluation of Pending Virtual Interrupts 


If the “virtual-interrupt delivery” VM-execution control is 1, certain actions cause a logical processor to evaluate 
pending virtual interrupts. 


The following actions cause the evaluation of pending virtual interrupts: VM entry; TPR virtualization; EOI virtual- 
ization; self-IPI virtualization; and posted-interrupt processing. See Section 25.3.2.5, Section 28.1.2, Section 
28.1.4, Section 28.1.5, and Section 28.6 for details of when evaluation of pending virtual interrupts is performed. 
No other operations cause the evaluation of pending virtual interrupts, even if they modify RVI or VPPR. 


Evaluation of pending virtual interrupts uses the guest interrupt status (specifically, RVI; see Section 23.4.2). The 
following pseudocode details the evaluation of pending virtual interrupts: 
IF “interrupt-window exiting” is O AND 
RVI[7:4] > VPPR[7:4] (see Section 28.1.1 for definition of VPPR) 
THEN recognize a pending virtual interrupt; 
ELSE 
do not recognize a pending virtual interrupt; 
Fl; 


Once recognized, a virtual interrupt may be delivered in VMX non-root operation; see Section 28.2.2. 


Evaluation of pending virtual interrupts is caused only by VM entry, TPR virtualization, EOI virtualization, self-IPI 
virtualization, and posted-interrupt processing. No other operations do so, even if they modify RVI or VPPR. The 
logical processor ceases recognition of a pending virtual interrupt following the delivery of a virtual interrupt. 


28.2.2 _‘Virtual-Interrupt Delivery 


If a virtual interrupt has been recognized (see Section 28.2.1), it is delivered at an instruction boundary when the 
following conditions all hold: (1) RFLAGS.IF = 1; (2) there is no blocking by STI; (3) there is no blocking by MOV 
SS or by POP SS; and (4) the “interrupt-window exiting” VM-execution control is 0. 


Virtual-interrupt delivery has the same priority as that of VM exits due to the 1-setting of the “interrupt-window 
exiting” VM-execution control.2 Thus, non-maskable interrupts (NMIs) and higher priority events take priority over 
delivery of a virtual interrupt; delivery of a virtual interrupt takes priority over external interrupts and lower priority 
events. 


Virtual-interrupt delivery wakes a logical processor from the same inactive activity states as would an external 
interrupt. Specifically, it wakes a logical processor from the states entered using the HLT and MWAIT instructions. 
It does not wake a logical processor in the shutdown state or in the wait-for-SIPI state. 


Virtual-interrupt delivery updates the guest interrupt status (both RVI and SVI; see Section 23.4.2) and delivers an 
event within VMX non-root operation without a VM exit. The following pseudocode details the behavior of virtual- 
interrupt delivery (see Section 28.1.1 for definition of VISR, VIRR, and VPPR): 
Vector := RVI; 
VISR[Vector] := 1; 
SVI := Vector; 
VPPR := Vector & FOH; 
VIRR[Vector] := 0; 
IF any bits set in VIRR 
THEN RVI := highest index of bit set in VIRR 
ELSE RVI := 0; 
Fl; 
deliver interrupt with Vector through IDT; 


2. Alogical processor never recognizes or delivers a virtual interrupt if the “interrupt-window exiting” VM-execution control is 1. 
Because of this, the relative priority of virtual-interrupt delivery and VM exits due to the 1-setting of that control is not defined. 
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cease recognition of any pending virtual interrupt; 


If a logical processor is in enclave mode, an Asynchronous Enclave Exit (AEX) occurs before delivery of a virtual 
interrupt (see Chapter 35, “Enclave Exiting Events”). 


28.3. VIRTUALIZING CR8-BASED TPR ACCESSES 


In 64-bit mode, software can access the local APIC’s task-priority register (TPR) through CR8. Specifically, software 
uses the MOV from CR8 and MOV to CR8 instructions (see Section 10.8.6, “Task Priority in IA-32e Mode”). This 
section describes how these accesses can be virtualized. 


A virtual-machine monitor can virtualize these CR8-based APIC accesses by setting the “CR8-load exiting” and 
“CR8-store exiting” VM-execution controls, ensuring that the accesses cause VM exits (see Section 24.1.3). Alter- 
natively, there are methods for virtualizing some CR8-based APIC accesses without VM exits. 


Normally, an execution of MOV from CR8 or MOV to CR8 that does not fault or cause a VM exit accesses the APIC’s 
TPR. However, such an execution are treated specially if the “use TPR shadow” VM-execution control is 1. The 
following items provide details: 


¢ MOV from CR&8. The instruction loads bits 3:0 of its destination operand with bits 7:4 of VTPR (see Section 
28.1.1). Bits 63:4 of the destination operand are cleared. 


¢ MOV to CR8. The instruction stores bits 3:0 of its source operand into bits 7:4 of VTPR; the remainder of VTPR 
(bits 3:0 and bits 31:8) are cleared. Following this, the processor performs TPR virtualization (see Section 
28.1.2). 


28.4 VIRTUALIZING MEMORY-MAPPED APIC ACCESSES 


When the local APIC is in xAPIC mode, software accesses the local APIC’s control registers using a memory- 
mapped interface. Specifically, software uses linear addresses that translate to physical addresses on page frame 
indicated by the base address in the IA32_APIC_BASE MSR (see Section 10.4.4, “Local APIC Status and Location”). 
This section describes how these accesses can be virtualized. 


A virtual-machine monitor (VMM) can virtualize these memory-mapped APIC accesses by ensuring that any access 
to a linear address that would access the local APIC instead causes a VM exit. This could be done using paging or 
the extended page-table mechanism (EPT). Another way is by using the 1-setting of the “virtualize APIC accesses” 
VM-execution control. 


If the “virtualize APIC accesses” VM-execution control is 1, the logical processor treats specially memory accesses 
using linear addresses that translate to physical addresses in the 4-KByte APIC-access page.?’* (The APIC- 
access page is identified by the APIC-access address, a field in the VMCS; see Section 23.6.8.) 


In general, an access to the APIC-access page causes an APIC-access VM exit. APIC-access VM exits provide a 
VMM with information about the access causing the VM exit. Section 28.4.1 discusses the priority of APIC-access 
VM exits. 


Certain VM-execution controls enable the processor to virtualize certain accesses to the APIC-access page without 
a VM exit. In general, this virtualization causes these accesses to be made to the virtual-APIC page instead of the 
APIC-access page. 


3. Even when addresses are translated using EPT (see Section 27.2), the determination of whether an APIC-access VM exit occurs 
depends on an access's physical address, not its quest-physical address. Even when CRO.PG = 0, ordinary memory accesses by soft- 
ware use linear addresses; the fact that CRO.PG = O means only that the identity translation is used to convert linear addresses to 
physical (or guest-physical) addresses. 


4. If EPT is enabled and there is write to a quest-physical address that translates to an address on the APIC-access page that is eligible 
for sub-page write permissions (see Section 27.2.4.1), the processor may treat the write as if the “virtualize APIC accesses” VM-exe- 
cution control were O (and not apply the treatment specified in this section). For that reason, it is recommended that software not 
configure any guest-physical address that translates to an address on the APIC-access page to be eligible for sub-page write permis- 
sions. 
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NOTES 


Unless stated otherwise, this section characterizes only linear accesses to the APIC-access page; an 
access to the APIC-access page is a linear access if (1) it results from a memory access using a 
linear address; and (2) the access’s physical address is the translation of that linear address. 
Section 28.4.6 discusses accesses to the APIC-access page that are not linear accesses. 


The distinction between the APIC-access page and the virtual-APIC page allows a VMM to share 
paging structures or EPT paging structures among the virtual processors of a virtual machine (the 
shared paging structures referencing the same APIC-access address, which appears in the VMCS of 
all the virtual processors) while giving each virtual processor its own virtual APIC (the VMCS of each 
virtual processor will have a unique virtual-APIC address). 


Section 28.4.2 discusses when and how the processor may virtualize read accesses from the APIC-access page. 
Section 28.4.3 does the same for write accesses. When virtualizing a write to the APIC-access page, the processor 
typically takes actions in addition to passing the write through to the virtual-APIC page. 


The discussion in those sections uses the concept of an operation within which these memory accesses may occur. 
For those discussions, an “operation” can be an iteration of a REP-prefixed string instruction, an execution of any 
other instruction, or delivery of an event through the IDT. 


The 1-setting of the “virtualize APIC accesses” VM-execution control may also affect accesses to the APIC-access 
page that do not result directly from linear addresses. This is discussed in Section 28.4.6. 


Special treatment may apply to Intel SGX instructions or if the logical processor is in enclave mode. See Section 
37.5.3 for details. 


28.4.1 Priority of APIC-Access VM Exits 


The following items specify the priority of APIC-access VM exits relative to other events. 


° The priority of an APIC-access VM exit due to a memory access is below that of any page fault or EPT violation 
that that access may incur. That is, an access does not cause an APIC-access VM exit if it would cause a page 
fault or an EPT violation. 


¢ Amemory access does not cause an APIC-access VM exit until after the accessed flags are set in the paging 
structures (including EPT paging structures, if enabled). 


¢ Awrite access does not cause an APIC-access VM exit until after the dirty flags are set in the appropriate paging 
structure and EPT paging structure (if enabled). 


¢ With respect to all other events, any APIC-access VM exit due to a memory access has the same priority as any 
page fault or EPT violation that the access could cause. (This item applies to other events that the access may 
generate as well as events that may be generated by other accesses by the same operation.) 


These principles imply, among other things, that an APIC-access VM exit may occur during the execution of a 
repeated string instruction (including INS and OUTS). Suppose, for example, that the first n iterations (n may be 
0) of such an instruction do not access the APIC-access page and that the next iteration does access that page. As 
a result, the first n iterations may complete and be followed by an APIC-access VM exit. The instruction pointer 
saved in the VMCS references the repeated string instruction and the values of the general-purpose registers 
reflect the completion of n iterations. 


28.4.2 Virtualizing Reads from the APIC-Access Page 


A read access from the APIC-access page causes an APIC-access VM exit if any of the following are true: 
¢ The “use TPR shadow” VM-execution control is 0. 

¢ The access is for an instruction fetch. 

¢ The access is more than 32 bits in size. 


°® The access is part of an operation for which the processor has already virtualized a write to the APIC-access 
page. 


28-6 Vol. 3C 


APIC VIRTUALIZATION AND VIRTUAL INTERRUPTS 


° The access is not entirely contained within the low 4 bytes of a naturally aligned 16-byte region. That is, bits 
3:2 of the access’s address are 0, and the same is true of the address of the highest byte accessed. 


If none of the above are true, whether a read access is virtualized depends on the setting of the “APIC-register 
virtualization” VM-execution control: 


¢ If “APIC-register virtualization” and “virtual-interrupt delivery” VM-execution controls are both 0, a read access 
is virtualized if its page offset is O80H (task priority); otherwise, the access causes an APIC-access VM exit. 


¢ If the “APIC-register virtualization” VM-execution control is 0 and the “virtual-interrupt delivery” VM-execution 
control is 1, a read access is virtualized if its page offset is O80H (task priority), OBOH (end of interrupt), or 
300H (interrupt command — low); otherwise, the access causes an APIC-access VM exit. 


e =If “APIC-register virtualization” is 1, a read access is virtualized if it is entirely within one the following ranges 
of offsets: 


— 020H-023H (local APIC ID); 

— 030H-033H (local APIC version); 

— O80H-083H (task priority); 

— OBOH-OB3H (end of interrupt); 

— ODOH-OD3H (logical destination); 

— OEOH-OE3H (destination format); 

— OFOH-OF3H (spurious-interrupt vector); 


— 100H-103H, 110H-113H, 120H-123H, 130H-133H, 140H-143H, 150H-153H, 160H-163H, or 170H- 
173H (in-service); 


— 180H-183H, 190H-193H, 1AOH-1A3H, 1BOH-1B3H, 1COH-1C3H, 1DOH-1D3H, 1EQH-1E3H, or 1FOH- 
1F3H (trigger mode); 


— 200H-203H, 210H-213H, 220H-223H, 230H-233H, 240H-243H, 250H-253H, 260H-263H, or 270H- 
273H (interrupt request); 


— 280H-283H (error status); 

— 300H-303H or 310H-313H (interrupt command); 

— 320H-323H, 330H-333H, 340H-343H, 350H-353H, 360H-363H, or 370H-373H (LVT entries); 
— 380H-383H (initial count); or 

— 3EQOH-3E3H (divide configuration). 

In all other cases, the access causes an APIC-access VM exit. 


A read access from the APIC-access page that is virtualized returns data from the corresponding page offset on the 
virtual-APIC page.® 


28.4.3 ‘Virtualizing Writes to the APIC-Access Page 


Whether a write access to the APIC-access page is virtualized depends on the settings of the VM-execution controls 
and the page offset of the access. Section 28.4.3.1 details when APIC-write virtualization occurs. 


Unlike reads, writes to the local APIC have side effects; because of this, virtualization of writes to the APIC-access 
page may require emulation specific to the access’s page offset (which identifies the APIC register being accessed). 
Section 28.4.3.2 describes this APIC-write emulation. 


For some page offsets, it is necessary for software to complete the virtualization after a write completes. In these 
cases, the processor causes an APIC-write VM exit to invoke VMM software. Section 28.4.3.3 discusses APIC- 
write VM exits. 


5. The memory type used for accesses that read from the virtual-APIC page is reported in bits 53:50 of the IA32_VMX_BASIC MSR 
(see Appendix A.1). 
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28.4.3.1 Determining Whether a Write Access is Virtualized 

A write access to the APIC-access page causes an APIC-access VM exit if any of the following are true: 
° The “use TPR shadow” VM-execution control is 0. 

¢ The access is more than 32 bits in size. 


¢ The access is part of an operation for which the processor has already virtualized a write (with a different page 
offset or a different size) to the APIC-access page. 


¢ The access is not entirely contained within the low 4 bytes of a naturally aligned 16-byte region. That is, bits 
3:2 of the access’s address are 0, and the same is true of the address of the highest byte accessed. 


If none of the above are true, whether a write access is virtualized depends on the settings of the “APIC-register 
virtualization” and “virtual-interrupt delivery” VM-execution controls: 


¢ If the “APIC-register virtualization” and “virtual-interrupt delivery” VM-execution controls are both 0, a write 
access is virtualized if its page offset is O80H; otherwise, the access causes an APIC-access VM exit. 


°¢ If the “APIC-register virtualization” VM-execution control is 0 and the “virtual-interrupt delivery” VM-execution 
control is 1, a write access is virtualized if its page offset is O80H (task priority), OBOH (end of interrupt), and 
300H (interrupt command — low); otherwise, the access causes an APIC-access VM exit. 


° If the “APIC-register virtualization” VM-execution control is 1, a write access is virtualized if it is entirely within 
one the following ranges of offsets: 


— 020H-023H (local APIC ID); 

— O80H-083H (task priority); 

— OBOH-OB3H (end of interrupt); 

— ODOH-OD3H (logical destination); 

— OEOH-OE3H (destination format); 

— OFOH-OF3H (spurious-interrupt vector); 

— 280H-283H (error status); 

— 300H-303H or 310H-313H (interrupt command); 

— 320H-323H, 330H-333H, 340H-343H, 350H-353H, 360H-363H, or 370H-373H (LVT entries); 
— 380H-383H (initial count); or 

— 3E0H-3E3H (divide configuration). 

In all other cases, the access causes an APIC-access VM exit. 


The processor virtualizes a write access to the APIC-access page by writing data to the corresponding page offset 
on the virtual-APIC page.® Following this, the processor performs certain actions after completion of the operation 
of which the access was a part.” APIC-write emulation is described in Section 28.4.3.2. 


28.4.3.2 APIC-Write Emulation 


If the processor virtualizes a write access to the APIC-access page, it performs additional actions after completion 
of an operation of which the access was a part. These actions are called APIC-write emulation. 


The details of APIC-write emulation depend upon the page offset of the virtualized write access:® 


® O80H (task priority). The processor clears bytes 3:1 of VIPR and then causes TPR virtualization (Section 
28.1.2). 


6. The memory type used for accesses that write to the virtual-APIC page is reported in bits 53:50 of the IA32_VMX_BASIC MSR (see 
Appendix A.1). 


7. Recall that, for the purposes of this discussion, an operation is an iteration of a REP-prefixed string instruction, an execution of any 
other instruction, or delivery of an event through the IDT. 


8. For any operation, there can be only one page offset for which a write access was virtualized. This is because a write access is not 
virtualized if the processor has already virtualized a write access for the same operation with a different page offset. 
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®  OBOH (end of interrupt). If the “virtual-interrupt delivery” VM-execution control is 1, the processor clears VEOI 
and then causes EOI virtualization (Section 28.1.4); otherwise, the processor causes an APIC-write VM exit 
(Section 28.4.3.3). 


¢ =300H (interrupt command — low). If the “virtual-interrupt delivery” VM-execution control is 1, the processor 
checks the value of VICR_LO to determine whether the following are all true: 


— Reserved bits (31:20, 17:16, 13) and bit 12 (delivery status) are all 0. 
— Bits 19:18 (destination shorthand) are 01B (self). 

— Bit 15 (trigger mode) is 0 (edge). 

— Bits 10:8 (delivery mode) are OOOB (fixed). 

— Bits 7:4 (the upper half of the vector) are not OOOOB. 


If all of the items above are true, the processor performs self-IPI virtualization using the 8-bit vector in byte 0 
of VICR_LO (Section 28.1.5). 


If the “virtual-interrupt delivery” VM-execution control is 0, or if any of the items above are false, the 
processor causes an APIC-write VM exit (Section 28.4.3.3). 


¢ =310H-313H (interrupt command — high). The processor clears bytes 2:0 of VICR_HI. No other virtualization or 
VM exit occurs. 


¢ Any other page offset. The processor causes an APIC-write VM exit (Section 28.4.3.3). 


APIC-write emulation takes priority over system-management interrupts (SMIs), INIT signals, and lower priority 
events. APIC-write emulation is not blocked if RFLAGS.IF = 0 or by the MOV SS, POP SS, or STI instructions. 


If an operation causes a fault after a write access to the APIC-access page and before APIC-write emulation, and 
that fault is delivered without a VM exit, APIC-write emulation occurs after the fault is delivered and before the fault 
handler can execute. If an operation causes a VM exit (perhaps due to a fault) after a write access to the APIC- 
access page and before APIC-write emulation, the APIC-write emulation does not occur. 


28.4.3.3 APIC-Write VM Exits 


In certain cases, VMM software must be invoked to complete the virtualization of a write access to the APIC-access 
page. In this case, APIC-write emulation causes an APIC-write VM exit. (Section 28.4.3.2 details the cases that 
causes APIC-write VM exits. ) 


APIC-write VM exits are invoked by APIC-write emulation, and APIC-write emulation occurs after an operation that 
performs a write access to the APIC-access page. Because of this, every APIC-write VM exit is trap-like: it occurs 
after completion of the operation containing the write access that caused the VM exit (for example, the value of 
CS:RIP saved in the guest-state area of the VMCS references the next instruction). 


The basic exit reason for an APIC-write VM exit is “APIC write.” The exit qualification is the page offset of the write 
access that led to the VM exit. 


As noted in Section 28.5, execution of WRMSR with ECX = 83FH (self-IPI MSR) can lead to an APIC-write VM exit if 
the “virtual-interrupt delivery” VM-execution control is 1. The exit qualification for such an APIC-write VM exit is 
3FOH. 


28.4.4 —_Instruction-Specific Considerations 


Certain instructions that use linear address may cause page faults even though they do not use those addresses to 
access memory. The APIC-virtualization features may affect these instructions as well: 


¢ CLFLUSH, CLFLUSHOPT. With regard to faulting, the processor operates as if each of these instructions reads 
from the linear address in its source operand. If that address translates to one on the APIC-access page, the 
instruction may cause an APIC-access VM exit. If it does not, it will flush the corresponding cache line on the 
virtual-APIC page instead of the APIC-access page. 


¢ ENTER. With regard to faulting, the processor operates if ENTER writes to the byte referenced by the final 
value of the stack pointer (even though it does not if its size operand is non-zero). If that value translates to an 
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address on the APIC-access page, the instruction may cause an APIC-access VM exit. If it does not, it will cause 
the APIC-write emulation appropriate to the address’s page offset. 


MASKMOVQ and MASKMOVDQU. Even if the instruction’s mask is zero, the processor may operate with 
regard to faulting as if MASKMOVQ or MASKMOVDQU writes to memory (the behavior is implementation- 
specific). In such a situation, an APIC-access VM exit may occur. 


MONITOR. With regard to faulting, the processor operates as if MONITOR reads from the effective address in 
RAX. If the resulting linear address translates to one on the APIC-access page, the instruction may cause an 
APIC-access VM exit.? If it does not, it will monitor the corresponding address on the virtual-APIC page instead 
of the APIC-access page. 


PREFETCH. An execution of the PREFETCH instruction that would result in an access to the APIC-access page 
does not cause an APIC-access VM exit. Such an access may prefetch data; if so, it is from the corresponding 
address on the virtual-APIC page. 


Virtualization of accesses to the APIC-access page is principally intended for basic instructions such as AND, MOV, 
OR, TEST, XCHG, and XOR. Use of an instruction that normally operates on floating-point, SSE, AVX, or AVX-512 
registers may cause an APIC-access VM exit unconditionally regardless of the page offset it accesses on the APIC- 
access page. 


28.4.5 Issues Pertaining to Page Size and TLB Management 


The 1-setting of the “virtualize APIC accesses” VM-execution is guaranteed to apply only if translations to the APIC- 
access address use a 4-KByte page. The following items provide details: 


If EPT is not in use, any linear address that translates to an address on the APIC-access page should use a 4- 
KByte page. Any access to a linear address that translates to the APIC-access page using a larger page may 
operate as if the “virtualize APIC accesses” VM-execution control were 0. 


If EPT is in use, any guest-physical address that translates to an address on the APIC-access page should use a 
4-KByte page. Any access to a linear address that translates to a guest-physical address that in turn translates 
to the APIC-access page using a larger page may operate as if the “virtualize APIC accesses” VM-execution 
control were 0. (This is true also for guest-physical accesses to the APIC-access page; see Section 28.4.6.1.) 


In addition, software should perform appropriate TLB invalidation when making changes that may affect APIC- 
virtualization. The specifics depend on whether VPIDs or EPT is being used: 


VPIDs being used but EPT not being used. Suppose that there is a VPID that has been used before and that 
software has since made either of the following changes: (1) set the “virtualize APIC accesses” VM-execution 
control when it had previously been 0; or (2) changed the paging structures so that some linear address 
translates to the APIC-access address when it previously did not. In that case, software should execute 
INVVPID (see “INVVPID— Invalidate Translations Based on VPID” in Section 29.3) before performing on the 
same logical processor and with the same VPID.?° 


EPT being used. Suppose that there is an EPTP value that has been used before and that software has since 
made either of the following changes: (1) set the “virtualize APIC accesses” VM-execution control when it had 
previously been 0; or (2) changed the EPT paging structures so that some guest-physical address translates to 
the APIC-access address when it previously did not. In that case, software should execute INVEPT (see 
“INVEPT— Invalidate Translations Derived from EPT” in Section 29.3) before performing on the same logical 
processor and with the same EPTP value.!! 


Neither VPIDs nor EPT being used. No invalidation is required. 


This chapter uses the notation RAX, RIP, RSP, RFLAGS, etc. for processor registers because most processors that support VMX oper- 
ation also support Intel 64 architecture. For |A-32 processors, this notation refers to the 32-bit forms of those registers (EAX, EIP, 
ESP, EFLAGS, etc.). In a few places, notation such as EAX is used to refer specifically to lower 32 bits of the indicated register. 


10. INVVPID should use either (1) the all-contexts INVVPID type; (2) the single-context INVVPID type with the VPID in the INVVPID 


descriptor; or (3) the individual-address INVVPID type with the linear address and the VPID in the INVVPID descriptor. 


11. INVEPT should use either (1) the global INVEPT type; or (2) the single-context INVEPT type with the EPTP value in the INVEPT 


descriptor. 
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Failure to perform the appropriate TLB invalidation may result in the logical processor operating as if the “virtualize 
APIC accesses” VM-execution control were 0 in responses to accesses to the affected address. (No invalidation is 
necessary if neither VPIDs nor EPT is being used.) 


28.4.6 APIC Accesses Not Directly Resulting From Linear Addresses 


Section 28.4 has described the treatment of accesses that use linear addresses that translate to addresses on the 
APIC-access page. This section considers memory accesses that do not result directly from linear addresses. 


* An access is called a guest-physical access if (1) CRO.PG = 1;!2 (2) the “enable EPT” VM-execution control 
is 1;13 (3) the access’s physical address is the result of an EPT translation; and (4) either (a) the access was 
not generated by a linear address; or (b) the access’s guest-physical address is not the translation of the 
access’s linear address. Section 28.4.6.1 discusses the treatment of guest-physical accesses to the APIC- 
access page. 


¢ An access is called a physical access if (1) either (a) the “enable EPT” VM-execution control is 0; or (b) the 
access’s physical address is not the result of a translation through the EPT paging structures; and (2) either 
(a) the access is not generated by a linear address; or (b) the access’s physical address is not the translation 
of its linear address. Section 28.4.6.2 discusses the treatment of physical accesses to the APIC-access page. 


28.4.6.1 Guest-Physical Accesses to the APIC-Access Page 


Guest-physical accesses include the following when guest-physical addresses are being translated using EPT: 


¢ Reads from the guest paging structures when translating a linear address (Such an access uses a guest- 
physical address that is not the translation of that linear address). 


¢ Loads of the page-directory-pointer-table entries by MOV to CR when the logical processor is using (or that 
causes the logical processor to use) PAE paging (see Section 4.4). 


¢ Updates to the accessed and dirty flags in the guest paging structures when using a linear address (such an 
access uses a guest-physical address that is not the translation of that linear address). 


¢ Memory accesses by Intel Processor Trace when the “Intel PT uses guest physical addresses” VM-execution 
control is 1 (see Section 24.5.4). 


Every guest-physical access using a guest-physical address that translates to an address on the APIC-access page 
causes an APIC-access VM exit. Such accesses are never virtualized regardless of the page offset. 


The following items specify the priority relative to other events of APIC-access VM exits caused by guest-physical 
accesses to the APIC-access page. 


¢ The priority of an APIC-access VM exit caused by a guest-physical access to memory is below that of any EPT 
violation that that access may incur. That is, a guest-physical access does not cause an APIC-access VM exit if 
it would cause an EPT violation. 


¢ With respect to all other events, any APIC-access VM exit caused by a guest-physical access has the same 
priority as any EPT violation that the guest-physical access could cause. 


28.4.6.2 Physical Accesses to the APIC-Access Page 
Physical accesses include the following: 
¢ If the “enable EPT” VM-execution control is 0: 
— Reads from the paging structures when translating a linear address. 


— Loads of the page-directory-pointer-table entries by MOV to CR when the logical processor is using (or that 
causes the logical processor to use) PAE paging (see Section 4.4). 


12. If the capability MSR 1A32_VMX_CRO_FIXEDO reports that CRO.PG must be 1 in VMX operation, CRO.PG must be 1 unless the “unre- 
stricted guest” VM-execution control and bit 31 of the primary processor-based VM-execution controls are both 1. 


13. “Enable EPT” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution controls 
is 0, VMX non-root operation functions as if the “enable EPT” VM-execution control were O. See Section 23.6.2. 
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— Updates to the accessed and dirty flags in the paging structures. 


e If the “enable EPT” VM-execution control is 1, accesses to the EPT paging structures (including updates to the 
accessed and dirty flags for EPT). 


e Any of the following accesses made by the processor to support VMX non-root operation: 
— Accesses to the VMCS region. 


— Accesses to data structures referenced (directly or indirectly) by physical addresses in VM-execution control 
fields in the VMCS. These include the I/O bitmaps, the MSR bitmaps, and the virtual-APIC page. 


* Accesses that effect transitions into and out of SMM.!4 These include the following: 
— Accesses to SMRAM during SMI delivery and during execution of RSM. 
— Accesses during SMM VM exits (including accesses to MSEG) and during VM entries that return from SMM. 


A physical access to the APIC-access page may or may not cause an APIC-access VM exit. If it does not cause an 
APIC-access VM exit, it may access the APIC-access page or the virtual-APIC page. Physical write accesses to the 
APIC-access page may or may not cause APIC-write emulation or APIC-write VM exits. 


The priority of an APIC-access VM exit caused by physical access is not defined relative to other events that the 
access may cause. 


It is recommended that software not set the APIC-access address to any of the addresses used by physical memory 
accesses (identified above). For example, it should not set the APIC-access address to the physical address of any 
of the active paging structures if the “enable EPT” VM-execution control is 0. 


28.5 VIRTUALIZING MSR-BASED APIC ACCESSES 


When the local APIC is in x2APIC mode, software accesses the local APIC’s control registers using the MSR inter- 
face. Specifically, software uses the RDMSR and WRMSR instructions, setting ECX (identifying the MSR being 
accessed) to values in the range 800H-8FFH (see Section 10.12, “Extended XAPIC (x2APIC)”). This section 
describes how these accesses can be virtualized. 


A virtual-machine monitor can virtualize these MSR-based APIC accesses by configuring the MSR bitmaps (see 
Section 23.6.9) to ensure that the accesses cause VM exits (see Section 24.1.3). Alternatively, there are methods 
for virtualizing some MSR-based APIC accesses without VM exits. 


Normally, an execution of RDMSR or WRMSR that does not fault or cause a VM exit accesses the MSR indicated in 
ECX. However, such an execution treats some values of ECX in the range 800H-8FFH specially if the “virtualize 
x2APIC mode” VM-execution control is 1. The following items provide details: 


¢ RDMSR. The instruction’s behavior depends on the setting of the “APIC-register virtualization” VM-execution 
control. 


— Ifthe “APIC-register virtualization” VM-execution control is 0, behavior depends upon the value of ECX. 


¢ If ECX contains 808H (indicating the TPR MSR), the instruction reads the 8 bytes from offset O80H on 
the virtual-APIC page (VTPR and the 4 bytes above it) into EDX:EAX. This occurs even if the local APIC 
is not in x2APIC mode (no general-protection fault occurs because the local APIC is not x2APIC mode). 


¢ If ECX contains any other value in the range 800H-8FFH, the instruction operates normally. If the local 
APIC is in x2APIC mode and ECX indicates a readable APIC register, EDX and EAX are loaded with the 
value of that register. If the local APIC is not in x2APIC mode or ECX does not indicate a readable APIC 
register, a general-protection fault occurs. 


— If*APIC-register virtualization” is 1 and ECX contains a value in the range 800H-8FFH, the instruction reads 
the 8 bytes from offset X on the virtual-APIC page into EDX:EAX, where X = (ECX & FFH) « 4. This occurs 
even if the local APIC is not in x2APIC mode (no general-protection fault occurs because the local APIC is 
not in x2APIC mode). 


© WRMSR. The instruction’s behavior depends on the value of ECX and the setting of the “virtual-interrupt 
delivery” VM-execution control. 


14. Technically, these accesses do not occur in VMX non-root operation. They are included here for clarity. 
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Special processing applies in the following cases: (1) ECX contains 808H (indicating the TPR MSR); (2) ECX 
contains 80BH (indicating the EOI MSR) and the “virtual-interrupt delivery” VM-execution control is 1; and 
(3) ECX contains 83FH (indicating the self-IPI MSR) and the “virtual-interrupt delivery” VM-execution control 
is 1. 

If special processing applies, no general-protection exception is produced due to the fact that the local APIC is 
in xAPIC mode. However, WRMSR does perform the normal reserved-bit checking: 

— If ECX contains 808H or 83FH, a general-protection fault occurs if either EDX or EAX[31:8] is non-zero. 

— If ECX contains 80BH, a general-protection fault occurs if either EDX or EAX is non-zero. 


If there is no fault, WRMSR stores EDX:EAX at offset X on the virtual-APIC page, where X = (ECX & FFH) « 4. 
Following this, the processor performs an operation depending on the value of ECX: 

— If ECX contains 808H, the processor performs TPR virtualization (see Section 28.1.2). 

— If ECX contains 80BH, the processor performs EOI virtualization (see Section 28.1.4). 

— If ECX contains 83FH, the processor then checks the value of EAX[7:4] and proceeds as follows: 


¢ If the value is non-zero, the logical processor performs self-IPI virtualization with the 8-bit vector in 
EAX[7:0] (see Section 28.1.5). 


¢ If the value is zero, the logical processor causes an APIC-write VM exit as if there had been a write 
access to page offset 3FOH on the APIC-access page (see Section 28.4.3.3). 


If special processing does not apply, the instruction operates normally. If the local APIC is in x2APIC mode 
and ECX indicates a writable APIC register, the value in EDX:EAX is written to that register. If the local APIC is 
not in x2APIC mode or ECX does not indicate a writable APIC register, a general-protection fault occurs. 


28.6 POSTED-INTERRUPT PROCESSING 


Posted-interrupt processing is a feature by which a processor processes the virtual interrupts by recording them as 
pending on the virtual-APIC page. 


Posted-interrupt processing is enabled by setting the “process posted interrupts” VM-execution control. The 
processing is performed in response to the arrival of an interrupt with the posted-interrupt notification vector. 
In response to such an interrupt, the processor processes virtual interrupts recorded in a data structure called a 
posted-interrupt descriptor. The posted-interrupt notification vector and the address of the posted-interrupt 
descriptor are fields in the VMCS; see Section 23.6.8. 


If the “process posted interrupts” VM-execution control is 1, a logical processor uses a 64-byte posted-interrupt 
descriptor located at the posted-interrupt descriptor address. The posted-interrupt descriptor has the following 
format: 


Table 28-1. Format of Posted-Interrupt Descriptor 


Bit Name Description 

Position(s) 

255:0 Posted-interrupt requests One bit for each interrupt vector. There is a posted-interrupt request for a vector if 
the corresponding bit is 1 

256 Outstanding notification If this bit is set, there is a notification outstanding for one or more posted interrupts 
in bits 255:0 

511:257 Reserved for software and These bits may be used by software and by other agents in the system (e.g., 

other agents chipset). The processor does not modify these bits. 


The notation PIR (posted-interrupt requests) refers to the 256 posted-interrupt bits in the posted-interrupt 
descriptor. 


Use of the posted-interrupt descriptor differs from that of other data structures that are referenced by pointers in 
a VMCS. There is a general requirement that software ensure that each such data structure is modified only when 
no logical processor with a current VMCS that references it is in VMX non-root operation. That requirement does 
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not apply to the posted-interrupt descriptor. There is a requirement, however, that such modifications be done 
using locked read-modify-write instructions. 


If the “external-interrupt exiting” VM-execution control is 1, any unmasked external interrupt causes a VM exit (see 
Section 24.2). If the “process posted interrupts” VM-execution control is also 1, this behavior is changed and the 


processor handles an external interrupt as follows: 
1. 


7. 


15 
The local APIC is acknowledged; this provides the processor core with an interrupt vector, called here the 
physical vector. 


If the physical vector equals the posted-interrupt notification vector, the logical processor continues to the next 
step. Otherwise, a VM exit occurs as it would normally due to an external interrupt; the vector is saved in the 
VM-exit interruption-information field. 


The processor clears the outstanding-notification bit in the posted-interrupt descriptor. This is done atomically 
so as to leave the remainder of the descriptor unmodified (e.g., with a locked AND operation). 


The processor writes zero to the EOI register in the local APIC; this dismisses the interrupt with the posted- 
interrupt notification vector from the local APIC. 


The logical processor performs a logical-OR of PIR into VIRR and clears PIR. No other agent can read or write a 
PIR bit (or group of bits) between the time it is read (to determine what to OR into VIRR) and when it is cleared. 


The logical processor sets RVI to be the maximum of the old value of RVI and the highest index of all bits that 
were set in PIR; if no bit was set in PIR, RVI is left unmodified. 


The logical processor evaluates pending virtual interrupts as described in Section 28.2.1. 


The logical processor performs the steps above in an uninterruptible manner. If step #7 leads to recognition of a 
virtual interrupt, the processor may deliver that interrupt immediately. 


Steps #1 to #7 above occur when the interrupt controller delivers an unmasked external interrupt to the CPU core. 
The following items consider certain cases of interrupt delivery: 


Interrupt delivery can occur between iterations of a REP-prefixed instruction (after at least one iteration has 
completed but before all iterations have completed). If this occurs, the following items characterize processor 
state after posted-interrupt processing completes and before guest execution resumes: 


— RIP references the REP-prefixed instruction; 
— RCX, RSI, and RDI are updated to reflect the iterations completed; and 
— RFLAGS.RF = 1. 


Interrupt delivery can occur when the logical processor is in the active, HLT, or MWAIT states. If the logical 
processor had been in the active or MWAIT state before the arrival of the interrupt, it is in the active state 
following completion of step #7; if it had been in the HLT state, it returns to the HLT state after step #7 (ifa 
pending virtual interrupt was recognized, the logical processor may immediately wake from the HLT state). 


Interrupt delivery can occur while the logical processor is in enclave mode. If the logical processor had been in 
enclave mode before the arrival of the interrupt, an Asynchronous Enclave Exit (AEX) may occur before the 
steps #1 to #7 (see Chapter 35, “Enclave Exiting Events”). If no AEX occurs before step #1 and a VM exit 
occurs at step #2, an AEX occurs before the VM exit is delivered. 


15. VM entry ensures that the “process posted interrupts” VM-execution control is 1 only if the “external-interrupt exiting” VM-execu- 


tion control is also 1. SeeSection 25.2.1.1. 
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VMX INSTRUCTION REFERENCE 


NOTE 


This chapter was previously located in the Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 2B as chapter 5. 


29.1 OVERVIEW 


This chapter describes the virtual-machine extensions (VMX) for the Intel 64 and IA-32 architectures. VMX is 
intended to support virtualization of processor hardware and a system software layer acting as a host to multiple 
guest software environments. The virtual-machine extensions (VMX) includes five instructions that manage the 
virtual-machine control structure (VMCS), four instructions that manage VMX operation, two TLB-management 
instructions, and two instructions for use by guest software. Additional details of VMX are described in Chapter 22 
through Chapter 28. 


The behavior of the VMCS-maintenance instructions is summarized below: 


* VMPTRLD — This instruction takes a single 64-bit source operand that is in memory. It makes the referenced 
VMCS active and current, loading the current-VMCS pointer with this operand and establishes the current VMCS 
based on the contents of VMCS-data area in the referenced VMCS region. Because this makes the referenced 
VMCS active, a logical processor may start maintaining on the processor some of the VMCS data for the VMCS. 


* VMPTRST — This instruction takes a single 64-bit destination operand that is in memory. The current-VMCS 
pointer is stored into the destination operand. 


¢ VMCLEAR — This instruction takes a single 64-bit operand that is in memory. The instruction sets the launch 
state of the VMCS referenced by the operand to “clear”, renders that VMCS inactive, and ensures that data for 
the VMCS have been written to the VMCS-data area in the referenced VMCS region. If the operand is the same 
as the current-VMCS pointer, that pointer is made invalid. 


° VMREAD -— This instruction reads a component from a VMCS (the encoding of that field is given in a register 
operand) and stores it into a destination operand that may be a register or in memory. 


°© VMWRITE — This instruction writes a component to a VMCS (the encoding of that field is given in a register 
operand) from a source operand that may be a register or in memory. 


The behavior of the VMX management instructions is summarized below: 


® VMLAUNCH — This instruction launches a virtual machine managed by the VMCS. A VM entry occurs, trans- 
ferring control to the VM. 


® VMRESUME — This instruction resumes a virtual machine managed by the VMCS. A VM entry occurs, trans- 
ferring control to the VM. 


® VMXOFF — This instruction causes the processor to leave VMX operation. 


° VMXON — This instruction takes a single 64-bit source operand that is in memory. It causes a logical processor 
to enter VMX root operation and to use the memory referenced by the operand to support VMX operation. 


The behavior of the VMX-specific TLB-management instructions is summarized below: 


¢ INVEPT — This instruction invalidates entries in the TLBs and paging-structure caches that were derived from 
extended page tables (EPT). 


¢ INVVPID — This instruction invalidates entries in the TLBs and paging-structure caches based on a Virtual- 
Processor Identifier (VPID). 


None of the instructions above can be executed in compatibility mode; they generate invalid-opcode exceptions if 
executed in compatibility mode. 


The behavior of the guest-available instructions is summarized below: 


* VMCALL — This instruction allows software in VMX non-root operation to call the VMM for service. A VM exit 
occurs, transferring control to the VMM. 
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¢ = VMFUNC — This instruction allows software in VMX non-root operation to invoke a VM function (processor 
functionality enabled and configured by software in VMX root operation) without a VM exit. 


29.2 CONVENTIONS 


The operation sections for the VMX instructions in Section 29.3 use the pseudo-function VMexit, which indicates 
that the logical processor performs a VM exit. 


The operation sections also use the pseudo-functions VMsucceed, VMfail, VMfailInvalid, and VMfailValid. These 
pseudo-functions signal instruction success or failure by setting or clearing bits in RFLAGS and, in some cases, by 
writing the VM-instruction error field. The following pseudocode fragments detail these functions: 


V/Msucceed: 
CF :=0; 
PF :=0; 
AF :=0; 
ZF :=0; 
SF :=0; 
OF :=0; 


VMfail(ErrorNumber): 
IF VMCS pointer is valid 
THEN VMfailValid(ErrorNumber); 
ELSE VMfaillnvalid; 


Fl; 
VMfaillnvalid: 
CF :=1; 
PF :=0; 
AF :=0; 
ZF :=0; 
SF :=0; 
OF :=0; 
VMfailValid(ErrorNumber):// executed only if there is a current VMCS 
CF :=0; 
PF :=0; 
AF :=0; 
ZF := 1; 
SF :=0; 
OF :=0; 


Set the VM-instruction error field to ErrorNumber; 


The different VM-instruction error numbers are enumerated in Section 29.4, “VM Instruction Error Numbers”. 


29.3. VMX INSTRUCTIONS 


This section provides detailed descriptions of the VMX instructions. 
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INVEPT— Invalidate Translations Derived from EPT 


Opcode/ Op/En |Description 

Instruction 

66 OF 38 80 RM Invalidates EPT-derived entries in the TLBs and paging-structure caches (in 64-bit mode). 
INVEPT r64, m128 

66 OF 38 80 RM Invalidates EPT-derived entries in the TLBs and paging-structure caches (outside 64-bit 
INVEPT 132, m128 mode). 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
RM ModRM:reg (r) ModRM:r/m (r) NA NA 
Description 


Invalidates mappings in the translation lookaside buffers (TLBs) and paging-structure caches that were derived 
from extended page tables (EPT). (See Chapter 27, “VMX Support for Address Translation”.) Invalidation is based 
on the INVEPT type specified in the register operand and the INVEPT descriptor specified in the memory 
operand. 


Outside IA-32e mode, the register operand is always 32 bits, regardless of the value of CS.D; in 64-bit mode, the 
register operand has 64 bits (the instruction cannot be executed in compatibility mode). 


The INVEPT types supported by a logical processors are reported in the IA32_VMX_EPT_VPID_CAP MSR (see 
Appendix A, “VMX Capability Reporting Facility”). There are two INVEPT types currently defined: 


¢ Single-context invalidation. If the INVEPT type is 1, the logical processor invalidates all mappings associated 
with bits 51:12 of the EPT pointer (EPTP) specified in the INVEPT descriptor. It may invalidate other mappings 
as well. 


Global invalidation: If the INVEPT type is 2, the logical processor invalidates mappings associated with all 
EPTPs. 


If an unsupported INVEPT type is specified, the instruction fails. 


INVEPT invalidates all the specified mappings for the indicated EPTP(s) regardless of the VPID and PCID values with 
which those mappings may be associated. 


The INVEPT descriptor comprises 128 bits and contains a 64-bit EPTP value in bits 63:0 (see Figure 29-1). 


127 64 63 0 


Reserved (must be zero) EPT pointer (EPTP) 


Figure 29-1. INVEPT Descriptor 


Operation 


IF (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation 
THEN VM exit; 
ELSIF CPL > O 
THEN #GP(0); 
ELSE 


Vol. 3C 29-3 


VMX INSTRUCTION REFERENCE 


INVEPT_TYPE := value of register operand; 
IF IA32_VMX_EPT_VPID_CAP MSR indicates that processor does not support INVEPT_TYPE 
THEN VMfail(Invalid operand to INVEPT/INVVPID); 
ELSE // \NVEPT_TYPE must be 1 or 2 
INVEPT_DESC := value of memory operand; 
EPTP := INVEPT_DESC[63:0); 
CASE INVEPT_TYPE OF 
1: // single-context invalidation 
IF VM entry with the “enable EPT” VM execution control set to 1 
would fail due to the EPTP value 
THEN VMfail(Invalid operand to INVEPT/INVVPID); 


ELSE 
Invalidate mappings associated with EPTP[51:1 2]; 
VMsucceed; 
Fl; 
BREAK; 
2: // global invalidation 
Invalidate mappings associated with all EPTPs; 
VMsucceed; 
BREAK; 
ESAC; 
Fl; 
Fl; 
Flags Affected 


See the operation section and Section 29.2. 


Protected Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the memory operand effective address is outside the CS, DS, ES, FS, or GS segment limit. 


If the DS, ES, FS, or GS register contains an unusable segment. 
If the source operand is located in an execute-only code segment. 


#PF(fault-code) If a page fault occurs in accessing the memory operand. 


#SS(0) If the memory operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 
#UD If not in VMX operation. 


If the logical processor does not support EPT (IA32_VMX_PROCBASED_CTLS2[33]=0). 


If the logical processor supports EPT (IA32_VMX_PROCBASED_CTLS2[33]=1) but does not 
support the INVEPT instruction (IA32_VMX_EPT_VPID_CAP[20]=0). 


Real-Address Mode Exceptions 


#UD 


The INVEPT instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 


#UD 


The INVEPT instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 


#UD 


The INVEPT instruction is not recognized in compatibility mode. 
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64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the memory operand is in the CS, DS, ES, FS, or GS segments and the memory address is 
in a non-canonical form. 


#PF(fault-code) If a page fault occurs in accessing the memory operand. 


#SS(0) If the memory operand is in the SS segment and the memory address is in a non-canonical 
form. 
#UD If not in VMX operation. 


If the logical processor does not support EPT (IA32_VMX_PROCBASED_CTLS2[33]=0). 


If the logical processor supports EPT (IA32_VMX_PROCBASED_CTLS2[33]=1) but does not 
support the INVEPT instruction (IA32_VMX_EPT_VPID_CAP[20]=0). 
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INVVPID— Invalidate Translations Based on VPID 


Opcode/ Op/En Description 

Instruction 

66 OF 38 81 RM Invalidates entries in the TLBs and paging-structure caches based on VPID (in 64-bit 
INVVPID r64, m128 mode). 

66 OF 38 81 RM Invalidates entries in the TLBs and paging-structure caches based on VPID (outside 64-bit 
INVVPID r32, m128 mode). 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
RM ModRM:reg (r) ModRM:r/m (r) NA NA 
Description 


Invalidates mappings in the translation lookaside buffers (TLBs) and paging-structure caches based on virtual- 
processor identifier (VPID). (See Chapter 27, “VMX Support for Address Translation”.) Invalidation is based on 
the INVVPID type specified in the register operand and the INVVPID descriptor specified in the memory 
operand. 


Outside IA-32e mode, the register operand is always 32 bits, regardless of the value of CS.D; in 64-bit mode, the 
register operand has 64 bits (the instruction cannot be executed in compatibility mode). 


The INVVPID types supported by a logical processors are reported in the IA32_VMX_EPT_VPID_CAP MSR (see 
Appendix A, “VMX Capability Reporting Facility”). There are four INVVPID types currently defined: 


¢ Individual-address invalidation: If the INVVPID type is 0, the logical processor invalidates mappings for the 
linear address and VPID specified in the INVVPID descriptor. In some cases, it may invalidate mappings for 
other linear addresses (or other VPIDs) as well. 


¢ Single-context invalidation: If the INVVPID type is 1, the logical processor invalidates all mappings tagged with 
the VPID specified in the INVVPID descriptor. In some cases, it may invalidate mappings for other VPIDs as 
well. 


e = All-contexts invalidation: If the INVVPID type is 2, the logical processor invalidates all mappings tagged with all 
VPIDs except VPID OOOOH. In some cases, it may invalidate translations with VPID OOOOH as well. 


®  Single-context invalidation, retaining global translations: If the INVVPID type is 3, the logical processor 
invalidates all mappings tagged with the VPID specified in the INVVPID descriptor except global translations. In 
some cases, it may invalidate global translations (and mappings with other VPIDs) as well. See the “Caching 
Translation Information” section in Chapter 4 of the JA-32 Intel Architecture Software Developer’s Manual, 
Volumes 3A for information about global translations. 


If an unsupported INVVPID type is specified, the instruction fails. 


INVVPID invalidates all the specified mappings for the indicated VPID(s) regardless of the EPTP and PCID values 
with which those mappings may be associated. 


The INVVPID descriptor comprises 128 bits and consists of a VPID and a linear address as shown in Figure 29-2. 


127 6463 1615 0 


Linear Address Reserved (must be zero) | VPID 


Figure 29-2. INVVPID Descriptor 
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IF (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 


THEN #UD; 
ELSIF in VMX non-root operation 
THEN VM exit; 
ELSIF CPL > O 
THEN #GP(0); 
ELSE 


INVVPID_TYPE := value of register operand; 
IF IA32_VMX_EPT_VPID_CAP MSR indicates that processor does not support 


INVVPID_TYPE 


THEN VMfail(Invalid operand to INVEPT/INVVPID); 
ELSE // INVVPID_TYPE must be in the range 0-3 
INVVPID_DESC := value of memory operand; 
IF INVVPID_DESC[63:16] 4 0 
THEN VMfail(Invalid operand to INVEPT/INVVPID); 


ELSE 


CASE | 
0: 


NVVPID_TYPE OF 


// individual-address invalidation 
VPID := INVVPID_DESC[1 5:0]; 
IF VPID = 0 
THEN VMfail(Invalid operand to INVEPT/INVVPID); 
ELSE 
GL_ADDR := INVVPID_DESC[1 27:64]; 
IF (GL_ADDR is not in a canonical form) 


THEN 
VMfail(Invalid operand to INVEPT/INVVPID); 
ELSE 
Invalidate mappings for GL_ADDR tagged with VPID; 
VMsucceed; 
Fl; 
Fl; 
BREAK; 


// single-context invalidation 
VPID := INVVPID_DESC[1 5:0]; 
IF VPID = 0 
THEN VMfail(Invalid operand to INVEPT/INVVPID); 
ELSE 
Invalidate all mappings tagged with VPID; 
VMsucceed; 
Fl; 
BREAK; 
// all-context invalidation 
Invalidate all mappings tagged with all non-zero VPIDs; 
VMsucceed; 
BREAK; 
// single-context invalidation retaining globals 
VPID := INVVPID_DESC[1 5:0]; 
IF VPID = 0 
THEN VMfail(Invalid operand to INVEPT/INVVPID); 
ELSE 
Invalidate all mappings tagged with VPID except global translations; 
VMsucceed; 
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Fl; 


Fl; 
Fl; 


Flags Affected 


Fl; 
BREAK; 
ESAC; 


See the operation section and Section 29.2. 


Protected Mode Exceptions 


#GP(0) 


#PF(fault-code) 
#SS(0) 


#UD 


If the current privilege level is not 0. 

If the memory operand effective address is outside the CS, DS, ES, FS, or GS segment limit. 
If the DS, ES, FS, or GS register contains an unusable segment. 

If the source operand is located in an execute-only code segment. 

If a page fault occurs in accessing the memory operand. 

If the memory operand effective address is outside the SS segment limit. 

If the SS register contains an unusable segment. 

If not in VMX operation. 

If the logical processor does not support VPIDs (IA32_VMX_PROCBASED_CTLS2[37]=0). 


If the logical processor supports VPIDs (IA32_VMX_PROCBASED_CTLS2[37]=1) but does not 
support the INVVPID instruction (IA32_VMX_EPT_VPID_CAP[32]=0). 


Real-Address Mode Exceptions 


#UD 


The INVVPID instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 


#UD 


The INVVPID instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 


#UD 


The INVVPID instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 


#GP(0) 
#PF(fault-code) 
#SS(0) 


#UD 
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If the current privilege level is not 0. 


If the memory operand is in the CS, DS, ES, FS, or GS segments and the memory address is 
in a non-canonical form. 


If a page fault occurs in accessing the memory operand. 


If the memory destination operand is in the SS segment and the memory address is in a non- 
canonical form. 


If not in VMX operation. 
If the logical processor does not support VPIDs (IA32_VMX_PROCBASED_CTLS2[37]=0). 


If the logical processor supports VPIDs (IA32_VMX_PROCBASED_CTLS2[37]=1) but does not 
support the INVVPID instruction (IA32_VMX_EPT_VPID_CAP[32]=0). 
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VMCALL—Call to VM Monitor 


Opcode/ Op/En Description 

Instruction 

OF 01 C1 ZO Call to VM monitor by causing VM exit. 
VMCALL 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
ZO NA NA NA NA 
Description 


This instruction allows guest software can make a call for service into an underlying VM monitor. The details of the 
programming interface for such calls are VMM-specific; this instruction does nothing more than cause a VM exit, 
registering the appropriate exit reason. 


Use of this instruction in VMX root operation invokes an SMM monitor (see Section 30.15.2). This invocation will acti- 
vate the dual-monitor treatment of system-management interrupts (SMIs) and system-management mode (SMM) 
if it is not already active (see Section 30.15.6). 


Operation 


IF not in VMX operation 
THEN #UD; 
ELSIF in VMX non-root operation 
THEN VM exit; 
ELSIF (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF CPL > 0 
THEN #GP(0); 
ELSIF in SMM or the logical processor does not support the dual-monitor treatment of SMls and SMM or the valid bit in the 
IA32_SMM_MONITOR_CTL MSR is clear 
THEN VMfail (VMCALL executed in VMX root operation); 
ELSIF dual-monitor treatment of SMIs and SMM is active 
THEN perform an SMM VM exit (see Section 30.15.2); 
ELSIF current-VMCS pointer is not valid 
THEN VMfaillnvalid; 
ELSIF launch state of current VMCS is not clear 
THEN VMfailValid(VMCALL with non-clear VMCS); 
ELSIF VM-exit control fields are not valid (see Section 30.15.6.1) 
THEN VMfailValid (VMCALL with invalid VM-exit control fields); 
ELSE 
enter SMM; 
read revision identifier in MSEG; 
IF revision identifier does not match that supported by processor 
THEN 
leave SMM; 
VMfailValid(VMCALL with incorrect MSEG revision identifier); 
ELSE 
read SMM-monitor features field in MSEG (see Section 30.15.6.1); 
IF features field is invalid 
THEN 
leave SMM; 
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VMfailValid(VMCALL with invalid SMM-monitor features); 
ELSE activate dual-monitor treatment of SMIs and SMM (see Section 30.15.6); 
Fl; 
FI; 
FI; 


Flags Affected 
See the operation section and Section 29.2. 


Protected Mode Exceptions 
#GP(0) If the current privilege level is not 0 and the logical processor is in VMX root operation. 
#UD If executed outside VMX operation. 


Real-Address Mode Exceptions 
#UD If executed outside VMX operation. 


Virtual-8086 Mode Exceptions 
#UD If executed outside VMX non-root operation. 


Compatibility Mode Exceptions 
#UD If executed outside VMX non-root operation. 


64-Bit Mode Exceptions 
#UD If executed outside VMX operation. 
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Opcode/ Op/En Description 

Instruction 

66 OF C7 /6 M Copy VMCS data to VMCS region in memory. 

VMCLEAR m64 

Instruction Operand Encoding 
Op/En Operand 1 Operand 2 Operand 3 Operand 4 
M ModRM:r/m (r) NA NA NA 

Description 


This instruction applies to the VMCS whose VMCS region resides at the physical address contained in the instruction 
operand. The instruction ensures that VMCS data for that VMCS (some of these data may be currently maintained 
on the processor) are copied to the VMCS region in memory. It also initializes parts of the VMCS region (for 
example, it sets the launch state of that VMCS to clear). See Chapter 23, “Virtual-Machine Control Structures”. 


The operand of this instruction is always 64 bits and is always in memory. If the operand is the current-VMCS 
pointer, then that pointer is made invalid (set to FFFFFFFF_FFFFFFFFH). 


Note that the VMCLEAR instruction might not explicitly write any VMCS data to memory; the data may be already 
resident in memory before the VMCLEAR is executed. 


Operation 


IF (register operand) or (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 


THEN #UD; 


ELSIF in VMX non-root operation 
THEN VM exit; 


ELSIF CPL > 0 


THEN #GP(0); 


ELSE 


addr := contents of 64-bit in-memory operand; 
IF addr is not 4KB-aligned OR 
addr sets any bits beyond the physical-address width! 


THEN VMfail(VMCLEAR with invalid physical address); 


ELSIF addr = VMXON pointer 
THEN VMfail(VMCLEAR with VMXON pointer); 
ELSE 

ensure that data for VMCS referenced by the operand is in memory; 


Fl; 
Fl; 


Flags Affected 


initialize implementation-specific data in VMCS region; 
launch state of VMCS referenced by the operand := “clear” 


IF operand addr = current-VMCS pointer 


THEN current-VMCS pointer := FFFFFFFF_FFFFFFFFH; 


Fl; 
\V/Msucceed; 


See the operation section and Section 29.2. 


1. If [A32_VMX_BASIC[48] is read as 1, VMfail occurs if addr sets any bits in the range 63:32; see Appendix A.1. 
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Protected Mode Exceptions 

#GP(0) If the current privilege level is not 0. 
If the memory operand effective address is outside the CS, DS, ES, FS, or GS segment limit. 
If the DS, ES, FS, or GS register contains an unusable segment. 
If the operand is located in an execute-only code segment. 

#PF(fault-code) If a page fault occurs in accessing the memory operand. 


#SS(0) If the memory operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 
#UD If operand is a register. 


If not in VMX operation. 


Real-Address Mode Exceptions 
#UD The VMCLEAR instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 
#UD The VMCLEAR instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
#UD The VMCLEAR instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the source operand is in the CS, DS, ES, FS, or GS segments and the memory address is in 
a non-canonical form. 


#PF(fault-code) If a page fault occurs in accessing the memory operand. 


#SS(0) If the source operand is in the SS segment and the memory address is in a non-canonical 
form. 
#UD If operand is a register. 


If not in VMX operation. 
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VMFUNC—Invoke VM function 


Opcode/ Op/En Description 

Instruction 

NP OF 01 D4 ZO Invoke VM function specified in EAX. 
VMFUNC 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
ZO NA NA NA NA 
Description 


This instruction allows software in VMX non-root operation to invoke a VM function, which is processor functionality 
enabled and configured by software in VMX root operation. The value of EAX selects the specific VM function being 
invoked. 


The behavior of each VM function (including any additional fault checking) is specified in Section 24.5.6, 
“VM Functions”. 


Operation 
Perform functionality of the VM function specified in EAX; 


Flags Affected 
Depends on the VM function specified in EAX. See Section 24.5.6, “VM Functions”. 


Protected Mode Exceptions (not including those defined by specific VM functions) 
#UD If executed outside VMX non-root operation. 

If “enable VM functions” VM-execution control is 0. 

If EAX > 64. 


Real-Address Mode Exceptions 
Same exceptions as in protected mode. 


Virtual-8086 Exceptions 
Same exceptions as in protected mode. 


Compatibility Mode Exceptions 
Same exceptions as in protected mode. 


64-Bit Mode Exceptions 
Same exceptions as in protected mode. 
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VMLAUNCH/VMRESUME—Launch/Resume Virtual Machine 


Opcode/ Op/En Description 

Instruction 

OF 01 C2 ZO Launch virtual machine managed by current VMCS. 
VMLAUNCH 

OF 01 C3 ZO Resume virtual machine managed by current VMCS. 
VMRESUME 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
ZO NA NA NA NA 
Description 


Effects a VM entry managed by the current VMCS. 


¢ VMLAUNCH fails if the launch state of current VMCS is not “clear”. If the instruction is successful, it sets the 
launch state to “launched.” 


¢ VMRESUME fails if the launch state of the current VMCS is not “launched.” 


If VM entry is attempted, the logical processor performs a series of consistency checks as detailed in Chapter 25, 

“VM Entries”. Failure to pass checks on the VMX controls or on the host-state area passes control to the instruction 
following the VMLAUNCH or VMRESUME instruction. If these pass but checks on the guest-state area fail, the logical 
processor loads state from the host-state area of the VMCS, passing control to the instruction referenced by the RIP 
field in the host-state area. 


VM entry is not allowed when events are blocked by MOV SS or POP SS. Neither VMLAUNCH nor VMRESUME should 
be used immediately after either MOV to SS or POP to SS. 


Operation 


IF (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation 
THEN VMexit; 
ELSIF CPL > 0 
THEN #GP(0); 
ELSIF current-VMCS pointer is not valid 
THEN VMfaillnvalid; 
ELSIF events are being blocked by MOV SS 
THEN VMfailValid(VM entry with events blocked by MOV SS); 
ELSIF (VMLAUNCH and launch state of current VMCS is not “clear”) 
THEN VMfailValid(VMLAUNCH with non-clear VMCS); 
ELSIF (VMRESUME and launch state of current VMCS is not “launched") 
THEN VMfailValid(VMRESUME with non-launched VMCS); 
ELSE 
Check settings of VMX controls and host-state area; 
IF invalid settings 
THEN VMfailValid(VM entry with invalid VMX-control field(s)) or 
VMfailValid(VM entry with invalid host-state field(s)) or 
VMfailValid(VM entry with invalid executive-VMCS pointer)) or 
VMfailValid(VM entry with non-launched executive VMCS) or 
VMfailValid(VM entry with executive-VMCS pointer not VMXON pointer) or 
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Fl; 
Fl; 


VMfailValid(VM entry with invalid VM-execution control fields in executive 
VMCS) 
as appropriate; 
ELSE 

Attempt to load guest state and PDPTRs as appropriate; 

clear address-range monitoring; 

IF failure in checking guest state or PDPTRs 
THEN VM entry fails (see Section 25.8); 


ELSE 
Attempt to load MSRs from VM-entry MSR-load area; 
IF failure 
THEN VM entry fails 
(see Section 25.8); 
ELSE 
IF VMLAUNCH 
THEN launch state of VMCS := “launched”; 
Fl; 
IF in SMM and “entry to SMM” VM-entry control is 0 
THEN 
IF “deactivate dual-monitor treatment” VM-entry 
control is O 
THEN SMM-transfer VMCS pointer := 
current-VMCS pointer; 
Fl; 
IF executive-VMCS pointer is VMXON pointer 
THEN current-VMCS pointer := 
VMCS-link pointer; 
ELSE current-VMCS pointer := 
executive-VMCS pointer; 
Fl; 
leave SMM; 
Fl; 
VM entry succeeds; 
Fl; 


Fl; 


Further details of the operation of the VM-entry appear in Chapter 25. 


Flags Affected 
See the operation section and Section 29.2. 


Protected Mode Exceptions 


#GP(0) 
#UD 


If the current privilege level is not 0. 
If executed outside VMX operation. 


Real-Address Mode Exceptions 
The VMLAUNCH and VMRESUME instructions are not recognized in real-address mode. 


#UD 


Virtual-8086 Mode Exceptions 
The VMLAUNCH and VMRESUME instructions are not recognized in virtual-8086 mode. 


#UD 
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Compatibility Mode Exceptions 
#UD The VMLAUNCH and VMRESUME instructions are not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 
#UD If executed outside VMX operation. 
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VMPTRLD—Load Pointer to Virtual-Machine Control Structure 


Opcode/ Op/En Description 

Instruction 

NP OF C7 /6 M Loads the current VMCS pointer from memory. 
VMPTRLD m64 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
M ModRM:r/m (r) NA NA NA 
Description 


Marks the current-VMCS pointer valid and loads it with the physical address in the instruction operand. The instruc- 
tion fails if its operand is not properly aligned, sets unsupported physical-address bits, or is equal to the VMXON 
pointer. In addition, the instruction fails if the 32 bits in memory referenced by the operand do not match the VMCS 
revision identifier supported by this processor.! 


The operand of this instruction is always 64 bits and is always in memory. 


Operation 


IF (register operand) or (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation 
THEN VMexit; 
ELSIF CPL > 0 
THEN #GP(0); 
ELSE 
addr := contents of 64-bit in-memory source operand; 
IF addr is not 4KB-aligned OR 
addr sets any bits beyond the physical-address width? 
THEN VMfail(VMPTRLD with invalid physical address); 
ELSIF addr = VMXON pointer 
THEN VMfail(VMPTRLD with VMXON pointer); 
ELSE 
rev := 32 bits located at physical address addr; 
IF rev[30:0]  VMCS revision identifier supported by processor OR 
rev[31] = 1 AND processor does not support 1-setting of “VMCS shadowing” 
THEN VMfail(VMPTRLD with incorrect VMCS revision identifier); 
ELSE 
current-VMCS pointer := addr; 
VMsucceed; 
Fl; 
Fl; 
Fl; 


Flags Affected 


See the operation section and Section 29.2. 


1. Software should consult the VMX capability MSR VMX_BASIC to discover the VMCS revision identifier supported by this processor 
(see Appendix A, “VMX Capability Reporting Facility’). 


2. If l1A32_VMX_BASIC[48] is read as 1, VMfail occurs if addr sets any bits in the range 63:32; see Appendix A.1. 
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Protected Mode Exceptions 
#GP(0) If the current privilege level is not 0. 
If the memory source operand effective address is outside the CS, DS, ES, FS, or GS segment 
limit. 
If the DS, ES, FS, or GS register contains an unusable segment. 
If the source operand is located in an execute-only code segment. 
#PF(fault-code) If a page fault occurs in accessing the memory source operand. 


#SS(0) If the memory source operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 
#UD If operand is a register. 


If not in VMX operation. 


Real-Address Mode Exceptions 
#UD The VMPTRLD instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 
#UD The VMPTRLD instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
#UD The VMPTRLD instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the source operand is in the CS, DS, ES, FS, or GS segments and the memory address is in 
a non-canonical form. 


#PF(fault-code) If a page fault occurs in accessing the memory source operand. 


#SS(0) If the source operand is in the SS segment and the memory address is in a non-canonical 
form. 
#UD If operand is a register. 


If not in VMX operation. 
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VMPTRST—Store Pointer to Virtual-Machine Control Structure 


Opcode/ Op/En ‘| Description 

Instruction 

NP OF C7 /7 M Stores the current VMCS pointer into memory. 
VMPTRST m64 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
M ModRM:r/m (w) NA NA NA 
Description 


Stores the current-VMCS pointer into a specified memory address. The operand of this instruction is always 64 bits 
and is always in memory. 


Operation 


IF (register operand) or (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation 
THEN VMexit; 
ELSIF CPL > O 
THEN #GP(0); 
ELSE 
64-bit in-memory destination operand := current-VMCS pointer; 
VMsucceed; 
Fl; 


Flags Affected 
See the operation section and Section 29.2. 


Protected Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the memory destination operand effective address is outside the CS, DS, ES, FS, or GS 
segment limit. 


If the DS, ES, FS, or GS register contains an unusable segment. 
If the destination operand is located in a read-only data segment or any code segment. 
#PF(fault-code) If a page fault occurs in accessing the memory destination operand. 


#SS(0) If the memory destination operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 
#UD If operand is a register. 


If not in VMX operation. 


Real-Address Mode Exceptions 
#UD The VMPTRST instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 
#UD The VMPTRST instruction is not recognized in virtual-8086 mode. 
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Compatibility Mode Exceptions 
#UD The VMPTRST instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the destination operand is in the CS, DS, ES, FS, or GS segments and the memory address 
is in a non-canonical form. 


#PF(fault-code) If a page fault occurs in accessing the memory destination operand. 


#SS(0) If the destination operand is in the SS segment and the memory address is in a non-canonical 
form. 
#UD If operand is a register. 


If not in VMX operation. 
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VMREAD—Read Field from Virtual-Machine Control Structure 


Opcode/ Op/En Description 

Instruction 

NP OF 78 MR Reads a specified VMCS field (in 64-bit mode). 
VMREAD r/m64, r64 

NP OF 78 MR Reads a specified VMCS field (outside 64-bit mode). 
VMREAD r/m32, r32 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
MR ModRM:r/m (w) ModRM:reg (r) NA NA 
Description 


Reads a specified field from a VMCS and stores it into a specified destination operand (register or memory). In VMX 
root operation, the instruction reads from the current VMCS. If executed in VMX non-root operation, the instruction 
reads from the VMCS referenced by the VMCS link pointer field in the current VMCS. 


The VMCS field is specified by the VMCS-field encoding contained in the register source operand. Outside IA-32e 
mode, the source operand has 32 bits, regardless of the value of CS.D. In 64-bit mode, the source operand has 64 
bits. 


The effective size of the destination operand, which may be a register or in memory, is always 32 bits outside IA- 
32e mode (the setting of CS.D is ignored with respect to operand size) and 64 bits in 64-bit mode. If the VMCS field 
specified by the source operand is shorter than this effective operand size, the high bits of the destination operand 
are cleared to 0. If the VMCS field is longer, then the high bits of the field are not read. 


Note that any faults resulting from accessing a memory destination operand can occur only after determining, in 
the operation section below, that the relevant VMCS pointer is valid and that the specified VMCS field is supported. 


Operation 


IF (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation AND ("/MCS shadowing” is 0 OR source operand sets bits in range 63:15 OR 
VMREAD bit corresponding to bits 14:0 of source operand is 1)! 
THEN VMexit; 
ELSIF CPL > 0 
THEN #GP(0); 
ELSIF (in VMX root operation AND current-VMCS pointer is not valid) OR 
(in VMX non-root operation AND VMCS link pointer is not valid) 
THEN VMfaillnvalid; 
ELSIF source operand does not correspond to any VMCS field 
THEN VMfailValid(VMREAD/VMWRITE from/to unsupported VMCS component); 
ELSE 
IF in VMX root operation 
THEN destination operand := contents of field indexed by source operand in current VMCS; 
ELSE destination operand := contents of field indexed by source operand in VMCS referenced by VMCS link pointer; 
Fl; 
VMsucceed; 
Fl; 


1. The VMREAD bit for a source operand is defined as follows. Let x be the value of bits 14:0 of the source operand and let addr be the 
VMREAD-bitmap address. The corresponding VMREAD bit is in bit position x & 7 of the byte at physical address addr | (x » 3). 
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Flags Affected 
See the operation section and Section 29.2. 


Protected Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If a memory destination operand effective address is outside the CS, DS, ES, FS, or GS 
segment limit. 


If the DS, ES, FS, or GS register contains an unusable segment. 
If the destination operand is located in a read-only data segment or any code segment. 
#PF(fault-code) If a page fault occurs in accessing a memory destination operand. 


#SS(0) If a memory destination operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 
#UD If not in VMX operation. 


Real-Address Mode Exceptions 
#UD The VMREAD instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 
#UD The VMREAD instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
#UD The VMREAD instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the memory destination operand is in the CS, DS, ES, FS, or GS segments and the memory 
address is in a non-canonical form. 


#PF(fault-code) If a page fault occurs in accessing a memory destination operand. 


#SS(0) If the memory destination operand is in the SS segment and the memory address is in a non- 
canonical form. 
#UD If not in VMX operation. 
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VMRESUME—Resume Virtual Machine 


See VMLAUNCH/VMRESUME—Launch/Resume Virtual Machine. 
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VMWRITE—Write Field to Virtual-Machine Control Structure 


Opcode/ Op/En Description 

Instruction 

NP OF 79 RM Writes a specified VMCS field (in 64-bit mode). 
VMWRITE r64, r/m64 

NP OF 79 RM Writes a specified VMCS field (outside 64-bit mode). 
VMWRITE r32, r/m32 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
RM ModRM:reg (r) ModRM:r/m (r) NA NA 
Description 


Writes the contents of a primary source operand (register or memory) to a specified field in a VMCS. In VMX root 
operation, the instruction writes to the current VMCS. If executed in VMX non-root operation, the instruction writes 
to the VMCS referenced by the VMCS link pointer field in the current VMCS. 


The VMCS field is specified by the VMCS-field encoding contained in the register secondary source operand. 
Outside IA-32e mode, the secondary source operand is always 32 bits, regardless of the value of CS.D. In 64-bit 
mode, the secondary source operand has 64 bits. 


The effective size of the primary source operand, which may be a register or in memory, is always 32 bits outside 
IA-32e mode (the setting of CS.D is ignored with respect to operand size) and 64 bits in 64-bit mode. If the VMCS 
field specified by the secondary source operand is shorter than this effective operand size, the high bits of the 
primary source operand are ignored. If the VMCS field is longer, then the high bits of the field are cleared to 0. 


Note that any faults resulting from accessing a memory source operand occur after determining, in the operation 
section below, that the relevant VMCS pointer is valid but before determining if the destination VMCS field is 
supported. 


Operation 


IF (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation AND (“VMCS shadowing” is O OR secondary source operand sets bits in range 63:15 OR 
VMWRITE bit corresponding to bits 14:0 of secondary source operand is 1)! 
THEN VMexit; 
ELSIF CPL > 0 
THEN #GP(0); 
ELSIF (in VMX root operation AND current-VMCS pointer is not valid) OR 
(in VMX non-root operation AND VMCS-link pointer is not valid) 
THEN VMfaillnvalid; 
ELSIF secondary source operand does not correspond to any VMCS field 
THEN VMfailValid(VMREAD/VMWRITE from/to unsupported VMCS component); 
ELSIF VMCS field indexed by secondary source operand is a VM-exit information field AND 
processor does not support writing to such fields@ 
THEN VMfailValid(VMWRITE to read-only VMCS component); 
ELSE 


1. The VMWRITE bit for a secondary source operand is defined as follows. Let x be the value of bits 14:0 of the secondary source oper- 
and and let addr be the VMWRITE-bitmap address. The corresponding VMWRITE bit is in bit position x & 7 of the byte at physical 
address addr | (x » 3). 


2. Software can discover whether these fields can be written by reading the VMX capability MSR IA32_VMX_MISC (see Appendix A.6). 
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IF in VMX root operation 


THEN field indexed by secondary source operand in current VMCS := primary source operand; 
ELSE field indexed by secondary source operand in VMCS referenced by VMCS link pointer := primary source operand; 


Fl; 
\V/Msucceed; 
Fl; 
Flags Affected 


See the operation section and Section 29.2. 


Protected Mode Exceptions 
#GP(0) If the current privilege level is not 0. 
If a memory source operand effective address is outside the CS, DS, ES, FS, or GS segment 
limit. 
If the DS, ES, FS, or GS register contains an unusable segment. 
If the source operand is located in an execute-only code segment. 
#PF(fault-code) If a page fault occurs in accessing a memory source operand. 


#SS(0) If a memory source operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 
#UD If not in VMX operation. 


Real-Address Mode Exceptions 
#UD The VMWRITE instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 
#UD The VMWRITE instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
#UD The VMWRITE instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If the current privilege level is not 0. 


If the memory source operand is in the CS, DS, ES, FS, or GS segments and the memory 
address is in a non-canonical form. 


#PF(fault-code) If a page fault occurs in accessing a memory source operand. 


#SS(0) If the memory source operand is in the SS segment and the memory address is in a non- 
canonical form. 
#UD If not in VMX operation. 
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VMXOFF—Leave VMX Operation 


Opcode/ Op/En ‘| Description 
Instruction 

OF 01 C4 ZO Leaves VMX operation. 
VMXOFF 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Operand 4 
ZO NA NA NA NA 
Description 


Takes the logical processor out of VMX operation, unblocks INIT signals, conditionally re-enables A20M, and clears 
any address-range monitoring.! 


Operation 


IF (not in VMX operation) or (CRO.PE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 
ELSIF in VMX non-root operation 
THEN VMexit; 
ELSIF CPL > O 
THEN #GP(0); 
ELSIF dual-monitor treatment of SMIs and SMM is active 
THEN VMfail(VMXOFF under dual-monitor treatment of SMIs and SMM); 
ELSE 
leave VMX operation; 
unblock INIT; 
IF |A32_SMM_MONITOR_CTL[2] = 02 
THEN unblock SMls; 
IF outside SMX operation? 
THEN unblock and enable A20M; 


Fl; 
clear address-range monitoring; 
VMsucceed; 

Fl; 

Flags Affected 


See the operation section and Section 29.2. 


Protected Mode Exceptions 
#GP(0) If executed in VMX root operation with CPL > 0. 


1. See the information on MONITOR/MWAIT in Chapter 8, “Multiple-Processor Management,” of the Intel® 64 and IA-32 Architectures 
Software Developer's Manual, Volume 3A. 

2. Setting |A32_SMM_MONITOR_CTLI[bit 2] to 1 prevents VMXOFF from unblocking SMIs regardless of the value of the register’s value 
bit (bit 0). Not all processors allow this bit to be set to 1. Software should consult the VMX capability MSR IA32_VMX_MISC (see 
Appendix A.6) to determine whether this is allowed. 

3. A logical processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the 
last execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference.” 


29-26 Vol. 3C 


VMX INSTRUCTION REFERENCE 


#UD If executed outside VMX operation. 


Real-Address Mode Exceptions 
#UD The VMXOFF instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 
#UD The VMXOFF instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
#UD The VMXOFF instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 
#GP(0) If executed in VMX root operation with CPL > 0. 
#UD If executed outside VMX operation. 


Vol. 3C 29-27 


VMX INSTRUCTION REFERENCE 


VMXON—Enter VMX Operation 


Opcode/ Op/En Description 

Instruction 

F3 OF C7 /6 M Enter VMX root operation. 

VMXON m64 

Instruction Operand Encoding 
Op/En Operand 1 Operand 2 Operand 3 Operand 4 
M ModRM:r/m (r) NA NA NA 

Description 


Puts the logical processor in VMX operation with no current VMCS, blocks INIT signals, disables A20M, and clears 


any address-range monitoring established by the MONITOR instruction. 


The operand of this instruction is a 4KB-aligned physical address (the VMXON pointer) that references the VMXON 
region, which the logical processor may use to support VMX operation. This operand is always 64 bits and is always 


in memory. 


Operation 


IF (register operand) or (CRO.PE = 0) or (CR4.VMXE = 0) or (RFLAGS.VM = 1) or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; 


ELSIF not in VMX operation 


THEN 


IF (CPL > O) or (in AZOM mode) or 


(the values of CRO and CR4 are not supported in VMX operation; see Section 22.8) or 


bit O (lock bit) of [A32_FEATURE_CONTROL MSR is clear) or 


( 
(in SMX operation® and bit 1 of IA32_FEATURE_CONTROL MSR is clear) or 
( 


outside SMX operation and bit 2 of IA32_FEATURE_CONTROL MSR is clear) 
THEN #GP(0); 


ELSE 
addr := contents of 64-bit in-memory source operand; 
IF addr is not 4KB-aligned or 
addr sets any bits beyond the physical-address width? 


THEN VMfaillnvalid; 


ELSE 


rev := 32 bits located at physical address addr; 


IF rev[30:0] # VMCS revision identifier supported by processor OR rev[31] = 1 


THEN VMfaillnvalid; 
ELSE 
current-VMCS pointer := FFFFFFFF_FFFFFFFFH; 
enter VMX operation; 
block INIT signals; 

block and disable A20M; 


1. See the information on MONITOR/MWAIT in Chapter 8, “Multiple-Processor Management,” of the Intel® 64 and IA-32 Architectures 


Software Developer's Manual, Volume 3A. 


2. Alogical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. A logi- 
cal processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the last 


execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference.” 
3. If IA32_VMX_BASIC[48] is read as 1, VMfaillnvalid occurs if addr sets any bits in the range 63:32; see Appendix A.1. 
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Fl; 


Fl; 
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clear address-range monitoring; 
IF the processor supports Intel PT but does not allow it to be used in VMX operation! 
THEN IA32_RTIT_CTL.TraceEn := 0; 
Fl; 
VMsucceed; 
Fl; 


ELSIF in VMX non-root operation 


THEN VMexit; 
ELSIF CPL > O 
THEN #GP(0); 


ELSE VMfail("VMXON executed in VMX root operation’); 


Fl; 


Flags Affected 


See the operation section and Section 29.2. 


Protected Mode Exceptions 


#GP(0) 


#PF(fault-code) 
#SS(0) 


#UD 


If executed outside VMX operation with CPL>0 or with invalid CRO or CR4 fixed bits. 

If executed in A2Z0M mode. 

If the memory source operand effective address is outside the CS, DS, ES, FS, or GS segment 
limit. 

If the DS, ES, FS, or GS register contains an unusable segment. 

If the source operand is located in an execute-only code segment. 


If the value of the IA32_FEATURE_CONTROL MSR does not support entry to VMX operation in 
the current processor mode. 


If a page fault occurs in accessing the memory source operand. 

If the memory source operand effective address is outside the SS segment limit. 
If the SS register contains an unusable segment. 

If operand is a register. 

If executed with CR4.VMXE = 0. 


Real-Address Mode Exceptions 


#UD 


The VMXON instruction is not recognized in real-address mode. 


Virtual-8086 Mode Exceptions 


#UD 


The VMXON instruction is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 


#UD 


The VMXON instruction is not recognized in compatibility mode. 


64-Bit Mode Exceptions 


#GP(0) 


If executed outside VMX operation with CPL > 0 or with invalid CRO or CR4 fixed bits. 
If executed in A20M mode. 


If the source operand is in the CS, DS, ES, FS, or GS segments and the memory address is in 
a non-canonical form. 


1. Software should read the VMX capability MSR [A32_VMX_MISC to determine whether the processor allows Intel PT to be used in 
VMX operation (see Appendix A.6). 
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If the value of the IA32_FEATURE_CONTROL MSR does not support entry to VMX operation in 
the current processor mode. 


#PF(fault-code) If a page fault occurs in accessing the memory source operand. 


#SS(0) If the source operand is in the SS segment and the memory address is in a non-canonical 
form. 
#UD If operand is a register. 


If executed with CR4.VMXE = 0. 
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VM INSTRUCTION ERROR NUMBERS 


For certain error conditions, the VM-instruction error field is loaded with an error number to indicate the source of 
the error. Table 29-1 lists VM-instruction error numbers. 


Table 29-1. VM-Instruction Error Numbers 


Sa ae Description 

1 VMCALL executed in VMX root operation 

2 VMCLEAR with invalid physical address 

3 VMCLEAR with VMXON pointer 

4 VMLAUNCH with non-clear VMCS 

5 VMRESUME with non-launched VMCS 

6 VMRESUME after VMXOFF (VMXOFF and VMXON between VMLAUNCH and VMRESUME)? 
7 VM entry with invalid control field(s)“ 
[8 |VMentrywithinvalidhost-state field(s” 
9 VMPTRLD with invalid physical address 

10 VMPTRLD with VMXON pointer 

11 VMPTRLD with incorrect VMCS revision identifier 

12 VMREAD/VMWRITE from/to unsupported VMCS component 

13 VMWRITE to read-only VMCS component 

15 VMXON executed in VMX root operation 

16 VM entry with invalid executive-VMCS pointer® 


17 VM entry with non-launched executive vMcs® 


18 VM entry with executive-VMCS pointer not VMXON pointer (when attempting to deactivate the dual-monitor treatment of 
SMIs and SMM)? 

19 VMCALL with non-clear VMCS (when attempting to activate the dual-monitor treatment of SMls and SMM) 

20 VMCALL with invalid VM-exit control fields 

22 VMCALL with incorrect MSEG revision identifier (when attempting to activate the dual-monitor treatment of SMIs and SMM) 

23 VMXOFF under dual-monitor treatment of SMIs and SMM 

24 VMCALL with invalid SMM-monitor features (when attempting to activate the dual-monitor treatment of SMIs and SMM) 

25 VM entry with invalid VM-execution control fields in executive VMCS (when attempting to return from SMM) 

26 VM entry with events blocked by MOV SS. 

28 Invalid operand to INVEPT/INVVPID. 

NOTES: 


a. Earlier versions of this manual described this error as “/MRESUME with a corrupted VMCS". 

b. VM-entry checks on control fields and host-state fields may be performed in any order. Thus, an indication by error number of one 
cause does not imply that there are not also other errors. Different processors may give different error numbers for the same VMCS. 

c. Error number 7 is not used for VM entries that return from SMM that fail due to invalid VM-execution control fields in the executive 
VMCS. Error number 25 is used for these cases. 
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CHAPTER 30 
SYSTEM MANAGEMENT MODE 


This chapter describes aspects of IA-64 and IA-32 architecture used in system management mode (SMM). 


SMM provides an alternate operating environment that can be used to monitor and manage various system 
resources for more efficient energy usage, to control system hardware, and/or to run proprietary code. It was 
introduced into the IA-32 architecture in the Intel386 SL processor (a mobile specialized version of the Intel386 
processor). It is also available in the Pentium M, Pentium 4, Intel Xeon, P6 family, and Pentium and Intel486 
processors (beginning with the enhanced versions of the Intel486 SL and Intel486 processors). 


30.1 SYSTEM MANAGEMENT MODE OVERVIEW 


SMM is a special-purpose operating mode provided for handling system-wide functions like power management, 
system hardware control, or proprietary OEM-designed code. It is intended for use only by system firmware, not by 
applications software or general-purpose systems software. The main benefit of SMM is that it offers a distinct and 
easily isolated processor environment that operates transparently to the operating system or executive and soft- 
ware applications. 


When SMM is invoked through a system management interrupt (SMI), the processor saves the current state of the 
processor (the processor’s context), then switches to a separate operating environment defined by a new address 
space. The system management software executive (SMI handler) starts execution in that environment, and the 
critical code and data of the SMI handler reside in a physical memory region (GMRAM) within that address space. 
While in SMM, the processor executes SMI handler code to perform operations such as powering down unused disk 
drives or monitors, executing proprietary code, or placing the whole system in a suspended state. When the SMI 
handler has completed its operations, it executes a resume (RSM) instruction. This instruction causes the processor 
to reload the saved context of the processor, switch back to protected or real mode, and resume executing the 
interrupted application or operating-system program or task. 


The following SMM mechanisms make it transparent to applications programs and operating systems: 
¢ The only way to enter SMM is by means of an SMI. 


¢ The processor executes SMM code in a separate address space that can be made inaccessible from the other 
operating modes. 


¢ Upon entering SMM, the processor saves the context of the interrupted program or task. 
¢ All interrupts normally handled by the operating system are disabled upon entry into SMM. 
¢ The RSM instruction can be executed only in SMM. 


Section 30.3 describes transitions into and out of SMM. The execution environment after entering SMM is in real- 
address mode with paging disabled (CRO.PE = CRO.PG = 0). In this initial execution environment, the SMI handler 
can address up to 4 GBytes of memory and can execute all I/O and system instructions. Section 30.5 describes in 
detail the initial SMM execution environment for an SMI handler and operation within that environment. The SMI 
handler may subsequently switch to other operating modes while remaining in SMM. 


NOTES 


Software developers should be aware that, even if a logical processor was using the physical- 
address extension (PAE) mechanism (introduced in the P6 family processors) or was in IA-32e 
mode before an SMI, this will not be the case after the SMI is delivered. This is because delivery of 
an SMI disables paging (see Table 30-4). (This does not apply if the dual-monitor treatment of SMIs 
and SMM is active; see Section 30.15.) 
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30.1.1 System Management Mode and VMX Operation 


Traditionally, SMM services system management interrupts and then resumes program execution (back to the soft- 
ware stack consisting of executive and application software; see Section 30.2 through Section 30.13). 


A virtual machine monitor (VMM) using VMX can act as a host to multiple virtual machines and each virtual machine 
can support its own software stack of executive and application software. On processors that support VMX, virtual- 
machine extensions may use system-management interrupts (SMIs) and system-management mode (SMM) in one 
of two ways: 


° Default treatment. System firmware handles SMIs. The processor saves architectural states and critical 
states relevant to VMX operation upon entering SMM. When the firmware completes servicing SMIs, it uses 
RSM to resume VMX operation. 


¢ Dual-monitor treatment. Two VM monitors collaborate to control the servicing of SMIs: one VMM operates 
outside of SMM to provide basic virtualization in support for guests; the other VMM operates inside SMM (while 
in VMX operation) to support system-management functions. The former is referred to as executive monitor, 
the latter SMM-transfer monitor (STM).! 


The default treatment is described in Section 30.14, “Default Treatment of SMIs and SMM with VMX Operation and 
SMX Operation”. Dual-monitor treatment of SMM is described in Section 30.15, “Dual-Monitor Treatment of SMIs 
and SMM”, 


30.2 SYSTEM MANAGEMENT INTERRUPT (SMI) 


The only way to enter SMM is by signaling an SMI through the SMI# pin on the processor or through an SMI 
message received through the APIC bus. The SMI is a nonmaskable external interrupt that operates independently 
from the processor's interrupt- and exception-handling mechanism and the local APIC. The SMI takes precedence 
over an NMI and a maskable interrupt. SMM is non-reentrant; that is, the SMI is disabled while the processor is in 
SMM. 


NOTES 


In the Pentium 4, Intel Xeon, and P6 family processors, when a processor that is designated as an 
application processor during an MP initialization sequence is waiting for a startup IPI (SIPI), it is in 
a mode where SMIs are masked. However if a SMI is received while an application processor is in 
the wait for SIPI mode, the SMI will be pended. The processor then responds on receipt of a SIPI by 
immediately servicing the pended SMI and going into SMM before handling the SIPI. 


An SMI may be blocked for one instruction following execution of STI, MOV to SS, or POP into SS. 


30.3. SWITCHING BETWEEN SMM AND THE OTHER 
PROCESSOR OPERATING MODES 


Figure 2-3 shows how the processor moves between SMM and the other processor operating modes (protected, 
real-address, and virtual-8086). Signaling an SMI while the processor is in real-address, protected, or virtual-8086 
modes always causes the processor to switch to SMM. Upon execution of the RSM instruction, the processor always 
returns to the mode it was in when the SMI occurred. 


30.3.1 Entering SMM 


The processor always handles an SMI on an architecturally defined “interruptible” point in program execution 
(which is commonly at an IA-32 architecture instruction boundary). When the processor receives an SMI, it waits 
for all instructions to retire and for all stores to complete. The processor then saves its current context in SMRAM 
(see Section 30.4), enters SMM, and begins to execute the SMI handler. 


1. The dual-monitor treatment may not be supported by all processors. Software should consult the VMX capability MSR 
IA32_VMX_BASIC (see Appendix A.1) to determine whether it is supported. 


30-2 Vol. 3C 


SYSTEM MANAGEMENT MODE 


Upon entering SMM, the processor signals external hardware that SMI handling has begun. The signaling mecha- 
nism used is implementation dependent. For the P6 family processors, an SMI acknowledge transaction is gener- 
ated on the system bus and the multiplexed status signal EXF4 is asserted each time a bus transaction is generated 
while the processor is in SMM. For the Pentium and Intel486 processors, the SMIACT# pin is asserted. 


An SMI has a greater priority than debug exceptions and external interrupts. Thus, if an NMI, maskable hardware 
interrupt, or a debug exception occurs at an instruction boundary along with an SMI, only the SMI is handled. 
Subsequent SMI requests are not acknowledged while the processor is in SMM. The first SMI interrupt request that 
occurs while the processor is in SMM (that is, after SMM has been acknowledged to external hardware) is latched 
and serviced when the processor exits SMM with the RSM instruction. The processor will latch only one SMI while 
in SMM. 


See Section 30.5 for a detailed description of the execution environment when in SMM. 


30.3.2 Exiting From SMM 


The only way to exit SMM is to execute the RSM instruction. The RSM instruction is only available to the SMI 
handler; if the processor is not in SMM, attempts to execute the RSM instruction result in an invalid-opcode excep- 
tion (#UD) being generated. 


The RSM instruction restores the processor’s context by loading the state save image from SMRAM back into the 
processor's registers. The processor then returns an SMIACK transaction on the system bus and returns program 
control back to the interrupted program. 


NOTE 


On processors that support the shadow-stack feature, RSM loads the SSP register from the state 
save image in SMRAM (see Table 30-3). The value is made canonical by sign-extension before 
loading it into SSP. 


Upon successful completion of the RSM instruction, the processor signals external hardware that SMM has been 
exited. For the P6 family processors, an SMI acknowledge transaction is generated on the system bus and the 
multiplexed status signal EXF4 is no longer generated on bus cycles. For the Pentium and Intel486 processors, the 
SMIACT# pin is deserted. 


If the processor detects invalid state information saved in the SMRAM, it enters the shutdown state and generates 
a special bus cycle to indicate it has entered shutdown state. Shutdown happens only in the following situations: 


¢ Areserved bit in control register CR4 is set to 1 on a write to CR4. This error should not happen unless SMI 
handler code modifies reserved areas of the SMRAM saved state map (see Section 30.4.1). CR4 is saved in the 
state map in a reserved location and cannot be read or modified in its saved state. 


¢ An illegal combination of bits is written to control register CRO, in particular PG set to 1 and PE set to 0, or NW 
set to 1 and CD set to 0. 


° CR4.PCIDE would be set to 1 and IA32_EFER.LMA to O. 


¢ (For the Pentium and Intel486 processors only.) If the address stored in the SMBASE register when an RSM 
instruction is executed is not aligned on a 32-KByte boundary. This restriction does not apply to the P6 family 
processors. 


® CR4.CET would be set to 1 and CRO.WP to O. 


In the shutdown state, Intel processors stop executing instructions until a RESET#, INIT# or NMI# is asserted. 
While Pentium family processors recognize the SMI# signal in shutdown state, P6 family and Intel486 processors 
do not. Intel does not support using SMI# to recover from shutdown states for any processor family; the response 
of processors in this circumstance is not well defined. On Pentium 4 and later processors, shutdown will inhibit INTR 
and A20M but will not change any of the other inhibits. On these processors, NMIs will be inhibited if no action is 
taken in the SMI handler to uninhibit them (see Section 30.8). 


If the processor is in the HALT state when the SMI is received, the processor handles the return from SMM slightly 
differently (see Section 30.10). Also, the SMBASE address can be changed on a return from SMM (see Section 
30.11). 
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30.4 SMRAM 


Upon entering SMM, the processor switches to a new address space. Because paging is disabled upon entering 
SMM, this initial address space maps all memory accesses to the low 4 GBytes of the processor's physical address 
space. The SMI handler's critical code and data reside in a memory region referred to as system-management RAM 
(SMRAM). The processor uses a pre-defined region within SMRAM to save the processor's pre-SMI context. SMRAM 
can also be used to store system management information (such as the system configuration and specific informa- 
tion about powered-down devices) and OEM-specific information. 


The default SMRAM size is 64 KBytes beginning at a base physical address in physical memory called the SMBASE 
(see Figure 30-1). The SMBASE default value following a hardware reset is 30000H. The processor looks for the 
first instruction of the SMI handler at the address [SMBASE + 8000H]. It stores the processor’s state in the area 
from [SMBASE + FEQOH] to [SMBASE + FFFFH]. See Section 30.4.1 for a description of the mapping of the state 
save area. 


The system logic is minimally required to decode the physical address range for the SMRAM from [SMBASE + 
8000H] to [SMBASE + FFFFH]. A larger area can be decoded if needed. The size of this SMRAM can be between 32 
KBytes and 4 GBytes. 


The location of the SMRAM can be changed by changing the SMBASE value (see Section 30.11). It should be noted 
that all processors in a multiple-processor system are initialized with the same SMBASE value (30000H). Initializa- 
tion software must sequentially place each processor in SMM and change its SMBASE so that it does not overlap 
those of other processors. 


The actual physical location of the SMRAM can be in system memory or in a separate RAM memory. The processor 
generates an SMI acknowledge transaction (P6 family processors) or asserts the SMIACT# pin (Pentium and 
Intel486 processors) when the processor receives an SMI (see Section 30.3.1). 


System logic can use the SMI acknowledge transaction or the assertion of the SMIACT# pin to decode accesses to 
the SMRAM and redirect them (if desired) to specific SMRAM memory. If a separate RAM memory is used for 
SMRAM, system logic should provide a programmable method of mapping the SMRAM into system memory space 
when the processor is not in SMM. This mechanism will enable start-up procedures to initialize the SMRAM space 
(that is, load the SMI handler) before executing the SMI handler during SMM. 


30.4.1 SMRAM State Save Map 


When an IA-32 processor that does not support Intel 64 architecture initially enters SMM, it writes its state to the 
state save area of the SMRAM. The state save area begins at [SMBASE + 8000H + 7FFFH] and extends down to 
[SMBASE + 8000H + 7EOOH]. Table 30-1 shows the state save map. The offset in column 1 is relative to the 
SMBASE value plus 8000H. Reserved spaces should not be used by software. 


Some of the registers in the SMRAM state save area (marked YES in column 3) may be read and changed by the 
SMI handler, with the changed values restored to the processor registers by the RSM instruction. Some register 
images are read-only, and must not be modified (modifying these registers will result in unpredictable behavior). 
An SMI handler should not rely on any values stored in an area that is marked as reserved. 


SMRAM 


SMBASE + FFFFH 
Start of State Save Area 


SMI Handler Entry Point 
SMBASE + 8000H 


SMBASE 


Figure 30-1. SMRAM Usage 
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Offset Register Writable? 
(Added to SMBASE + 8000H) 
7FFCH CRO No 
7FF8H CR3 No 
7FF4H EFLAGS Yes 
7FFOH EIP Yes 
7FECH EDI Yes 
7FE8H ESI Yes 
7FE4H EBP Yes 
7FEOH ESP Yes 
7FDCH EBX Yes 
7FD8H EDX Yes 
7FD4H ECX Yes 
7FDOH EAX Yes 
7FCCH DR6 No 
7FC8H DR7 No 
7FC4H TR! No 
7FCOH Reserved No 
7FBCH cs! No 
7FB8H Fs! No 
7FB4H Ds! No 
7FBOH ss! No 
7FACH cs! No 
7FA8H es! No 
7FA4H I/O State Field, see Section 30.7 No 
7FAOH 1/0 Memory Address Field, see Section 30.7 No 
7F9FH-7F03H Reserved No 
7F02H Auto HALT Restart Field (Word) Yes 
7FOOH I/O Instruction Restart Field (Word) Yes 
7EFCH SMM Revision Identifier Field (Doubleword) No 
7EF8H SMBASE Field (Doubleword) Yes 
7€F7H - 7E00H Reserved No 


NOTE: 


1. The two most significant bytes are reserved. 


The following registers are saved (but not readable) and restored upon exiting SMM: 


* Control register CR4. (This register is cleared to all Os when entering SMM). 


° The hidden segment descriptor information stored in segment registers CS, DS, ES, FS, GS, and SS. 


If an SMI request is issued for the purpose of powering down the processor, the values of all reserved locations in 
the SMM state save must be saved to nonvolatile memory. 


The following state is not automatically saved and restored following an SMI and the RSM instruction, respectively: 
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* Debug registers DRO through DR3. 

¢ The x87 FPU registers. 

° The MTRRs. 

* Control register CR2. 

¢ The model-specific registers (for the P6 family and Pentium processors) or test registers TR3 through TR7 (for 
the Pentium and Intel486 processors). 

¢ The state of the trap controller. 

¢ The machine-check architecture registers. 

¢ The APIC internal interrupt state (ISR, IRR, etc.). 

¢ The microcode update state. 


If an SMI is used to power down the processor, a power-on reset will be required before returning to SMM, which 
will reset much of this state back to its default values. So an SMI handler that is going to trigger power down should 
first read these registers listed above directly, and save them (along with the rest of RAM) to nonvolatile storage. 
After the power-on reset, the continuation of the SMI handler should restore these values, along with the rest of 
the system's state. Anytime the SMI handler changes these registers in the processor, it must also save and restore 
them. 


NOTES 


A small subset of the MSRs (such as, the time-stamp counter and performance-monitoring 
counters) are not arbitrarily writable and therefore cannot be saved and restored. SMM-based 
power-down and restoration should only be performed with operating systems that do not use or 
rely on the values of these registers. 

Operating system developers should be aware of this fact and ensure that their operating-system 
assisted power-down and restoration software is immune to unexpected changes in these register 
values. 


30.4.1.1 SMRAM State Save Map and Intel 64 Architecture 


When the processor initially enters SMM, it writes its state to the state save area of the SMRAM. The state save area 
on an Intel 64 processor at [SMBASE + 8000H + 7FFFH] and extends to [SMBASE + 8000H + 7COOH]. 


Support for Intel 64 architecture is reported by CPUID.80000001:EDX[29] = 1. The layout of the SMRAM state save 
map is shown in Table 30-3. 


Additionally, the SMRAM state save map shown in Table 30-3 also applies to processors with the following CPUID 
signatures listed in Table 30-2, irrespective of the value in CPUID.80000001:EDX[29]. 


Table 30-2. Processor Signatures and 64-bit SMRAM State Save Map Format 


DisplayFamily_DisplayModel | Processor Families/Processor Number Series 

06_17H Intel Xeon Processor 5200, 5400 series, Intel Core 2 Quad processor Q9xxx, Intel Core 2 Duo 
processors E8000, T9000, 

06_OFH Intel Xeon Processor 3000, 3200, 5100, 5300, 7300 series, Intel Core 2 Quad, Intel Core 2 Extreme, 
Intel Core 2 Duo processors, Intel Pentium dual-core processors 

06_1CH 45 nm Intel® Atom™ processors 
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Offset Register Writable? 
(Added to SMBASE + 8000H) 
7FF8H CRO No 
7FFOH CR3 No 
7FE8H RFLAGS Yes 
7FEOH IA32_EFER Yes 
7FD8H RIP Yes 
7FDOH DR6 No 
7FC8H DR7 No 
7FC4H TR SEL! No 
7FCOH LDTR SEL! No 
7FBCH Gs seu! No 
7FB8H FS SEL! No 
7FB4H Ds SEL! No 
7FBOH SS SEL! No 
7FACH cs Se! No 
7FA8H ES SEL! No 
7FA4H IO_MISC No 
7F9CH IO_MEM_ADDR No 
7F94H RDI Yes 
7F8CH RSI Yes 
7F84H RBP Yes 
7F7CH RSP Yes 
7F74H RBX Yes 
7F6CH RDX Yes 
7F64H RCX Yes 
7F5CH RAX Yes 
7F54H R8 Yes 
7F4CH RO Yes 
7F44H R10 Yes 
7F3CH R11 Yes 
7F34H R12 Yes 
7F2CH R13 Yes 
7F24H R14 Yes 
7F1CH R15 Yes 
7F1BH-7FO4H Reserved No 
7F02H Auto HALT Restart Field (Word) Yes 
7FOOH I/O Instruction Restart Field (Word) Yes 
7EFCH SMM Revision Identifier Field (Doubleword) No 
7EF8H SMBASE Field (Doubleword) Yes 
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Table 30-3. SMRAM State Save Map for Intel 64 Architecture (Contd.) 


NOTE: 


1. The two most significant bytes are reserved. 


30.4.2 © SMRAM Caching 


An IA-32 processor does not automatically write back and invalidate its caches before entering SMM or before 


exiting SMM. Because of this behavior, care must be taken in the placement of the SMRAM in system memory and 


Offset Register Writable? 
(Added to SMBASE + 8000H) 
7EF7H - 7EE4H Reserved No 
7EEOH Setting of “enable EPT” VM-execution control No 
7€D8H Value of EPTP VM-execution control field No 
7€D7H - 7ECCOH Reserved No 
7EC8H SSP Yes 
7EC7H - 7EAOH Reserved No 
7E9CH LDT Base (lower 32 bits) No 
7€98H Reserved No 
7E94H IDT Base (lower 32 bits) No 
7€90H Reserved No 
7E8CH GDT Base (lower 32 bits) No 
7E8BH - 7E48H Reserved No 
7E40H CR4 (64 bits) No 
7€3FH - 7DFOH Reserved No 
7DE8H IO_RIP Yes 
7DE7H - 7DDCH Reserved No 
7DD8H IDT Base (Upper 32 bits) No 
7DD4H LDT Base (Upper 32 bits) No 
7DDOH GDT Base (Upper 32 bits) No 
7DCFH - 7COOH Reserved No 


in the caching of the SMRAM to prevent cache incoherence when switching back and forth between SMM and 


protected mode operation. Any of the following three methods of locating the SMRAM in system memory will guar- 


antee cache coherency. 


¢ Place the SMRAM in a dedicated section of system memory that the operating system and applications are 
prevented from accessing. Here, the SMRAM can be designated as cacheable (WB, WT, or WC) for optimum 


processor performance, without risking cache incoherence when entering or exiting SMM. 


¢ Place the SMRAM ina section of memory that overlaps an area used by the operating system (such as the video 
memory), but designate the SMRAM as uncacheable (UC). This method prevents cache access when in SMM to 


maintain cache coherency, but the use of uncacheable memory reduces the performance of SMM code. 


¢ Place the SMRAM in a section of system memory that overlaps an area used by the operating system and/or 


application code, but explicitly flush (write back and invalidate) the caches upon entering and exiting SMM 
mode. This method maintains cache coherency, but incurs the overhead of two complete cache flushes. 


For Pentium 4, Intel Xeon, and P6 family processors, a combination of the first two methods of locating the SMRAM 


is recommended. Here the SMRAM is split between an overlapping and a dedicated region of memory. Upon 


entering SMM, the SMRAM space that is accessed overlaps video memory (typically located in low memory). This 
SMRAM section is designated as UC memory. The initial SMM code then jumps to a second SMRAM section that is 
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located in a dedicated region of system memory (typically in high memory). This SMRAM section can be cached for 
optimum processor performance. 


For systems that explicitly flush the caches upon entering SMM (the third method described above), the cache flush 
can be accomplished by asserting the FLUSH# pin at the same time as the request to enter SMM (generally initi- 
ated by asserting the SMI# pin). The priorities of the FLUSH# and SMI¥# pins are such that the FLUSH# is serviced 
first. To guarantee this behavior, the processor requires that the following constraints on the interaction of FLUSH# 
and SMI# be met. In a system where the FLUSH# and SMI# pins are synchronous and the set up and hold times 
are met, then the FLUSH# and SMI# pins may be asserted in the same clock. In asynchronous systems, the 
FLUSH# pin must be asserted at least one clock before the SMI# pin to guarantee that the FLUSH# pin is serviced 
first. 


Upon leaving SMM (for systems that explicitly flush the caches), the WBINVD instruction should be executed prior 
to leaving SMM to flush the caches. 


NOTES 


In systems based on the Pentium processor that use the FLUSH# pin to write back and invalidate 
cache contents before entering SMM, the processor will prefetch at least one cache line in between 
when the Flush Acknowledge cycle is run and the subsequent recognition of SMI# and the assertion 
of SMIACT#. 

It is the obligation of the system to ensure that these lines are not cached by returning KEN# 
inactive to the Pentium processor. 


30.4.2.1 System Management Range Registers (SMRR) 


SMI handler code and data stored by SMM code resides in SMRAM. The SMRR interface is an enhancement in Intel 
64 architecture to limit cacheable reference of addresses in SMRAM to code running in SMM. The SMRR interface 
can be configured only by code running in SMM. Details of SMRR is described in Section 11.11.2.4. 


30.5 SMIHANDLER EXECUTION ENVIRONMENT 


Section 30.5.1 describes the initial execution environment for an SMI handler. An SMI handler may re-configure its 
execution environment to other supported operating modes. Section 30.5.2 discusses modifications an SMI 
handler can make to its execution environment. Section 30.5.3 discusses Control-flow Enforcement Technology 
(CET) interactions in the environment. 


30.5.1 Initial SMM Execution Environment 


After saving the current context of the processor, the processor initializes its core registers to the values shown in 
Table 30-4. Upon entering SMM, the PE and PG flags in control register CRO are cleared, which places the processor 
in an environment similar to real-address mode. The differences between the SMM execution environment and the 
real-address mode execution environment are as follows: 


° The addressable address space ranges from 0 to FFFFFFFFH (4 GBytes). 
¢ The normal 64-KByte segment limit for real-address mode is increased to 4 GBytes. 


° The default operand and address sizes are set to 16 bits, which restricts the addressable SMRAM address space 
to the 1-MByte real-address mode limit for native real-address-mode code. However, operand-size and 
address-size override prefixes can be used to access the address space beyond the 1-MByte. 


Table 30-4. Processor Register Initialization in SMM 


Register Contents 

General-purpose registers Undefined 

EFLAGS 00000002H 

EIP 00008000H 

CS selector SMM Base shifted right 4 bits (default 3000H) 
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Table 30-4. Processor Register Initialization in SMM 


CS base SMM Base (default 30000H) 

DS, ES, FS, GS, SS Selectors 0000H 

DS, ES, FS, GS, SS Bases 000000000H 

DS, ES, FS, GS, SS Limits OFFFFFFFFH 

CRO PE, EM, TS, and PG flags set to 0; others unmodified 
CR4 Cleared to zero 

DR6 Undefined 

DR7 00000400H 


¢ Near jumps and calls can be made to anywhere in the 4-GByte address space if a 32-bit operand-size override 
prefix is used. Due to the real-address-mode style of base-address formation, a far call or jump cannot transfer 
control to a segment with a base address of more than 20 bits (1 MByte). However, since the segment limit in 
SMM is 4 GBytes, offsets into a segment that go beyond the 1-MByte limit are allowed when using 32-bit 
operand-size override prefixes. Any program control transfer that does not have a 32-bit operand-size override 
prefix truncates the EIP value to the 16 low-order bits. 


* Data and the stack can be located anywhere in the 4-GByte address space, but can be accessed only with a 32- 
bit address-size override if they are located above 1 MByte. As with the code segment, the base address for a 
data or stack segment cannot be more than 20 bits. 


The value in segment register CS is automatically set to the default of 30000H for the SMBASE shifted 4 bits to the 
right; that is, 3000H. The EIP register is set to 8000H. When the EIP value is added to shifted CS value (the 
SMBASE), the resulting linear address points to the first instruction of the SMI handler. 


The other segment registers (DS, SS, ES, FS, and GS) are cleared to 0 and their segment limits are set to 4 GBytes. 
In this state, the SMRAM address space may be treated as a single flat 4-GByte linear address space. If a segment 
register is loaded with a 16-bit value, that value is then shifted left by 4 bits and loaded into the segment base 
(hidden part of the segment register). The limits and attributes are not modified. 


Maskable hardware interrupts, exceptions, NMI interrupts, SMI interrupts, A20M interrupts, single-step traps, 
breakpoint traps, and INIT operations are inhibited when the processor enters SMM. Maskable hardware interrupts, 
exceptions, single-step traps, and breakpoint traps can be enabled in SMM if the SMM execution environment 
provides and initializes an interrupt table and the necessary interrupt and exception handlers (see Section 30.6). 


30.5.2 SMI Handler Operating Mode Switching 


Within SMM, an SMI handler may change the processor's operating mode (e.g., to enable PAE paging, enter 64-bit 
mode, etc.) after it has made proper preparation and initialization to do so. For example, if switching to 32-bit 
protected mode, the SMI handler should follow the guidelines provided in Chapter 9, “Processor Management and 
Initialization”. If the SMI handler does wish to change operating mode, it is responsible for executing the appro- 
priate mode-transition code after each SMI. 


It is recommended that the SMI handler make use of all means available to protect the integrity of its critical code 
and data. In particular, it should use the system-management range register (SMRR) interface if it is available (see 
Section 11.11.2.4). The SMRR interface can protect only the first 4 GBytes of the physical address space. The SMI 
handler should take that fact into account if it uses operating modes that allow access to physical addresses beyond 
that 4-GByte limit (e.g. PAE paging or 64-bit mode). 


Execution of the RSM instruction restores the pre-SMI processor state from the SMRAM state-state map (see 
Section 30.4.1) into which it was stored when the processor entered SMM. (The SMBASE field in the SMRAM state- 
save map does not determine the state following RSM but rather the initial environment following the next entry to 
SMM.) Any required change to operating mode is performed by the RSM instruction; there is no need for the SMI 
handler to change modes explicitly prior to executing RSM. 
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30.5.3. Control-flow Enforcement Technology Interactions 


On processors that support CET shadow stacks, when the processor enters SMM, the processor saves the SSP 
register to the SMRAM state save area (see Table 30-3) and clears CR4.CET to 0. Thus, the initial execution envi- 
ronment of the SMI handler has CET disabled and all of the CET state of the interrupted program is still in the 
machine. An SMM that uses CET is required to save the interrupted program’s CET state and restore the CET state 
prior to exiting SMM. 


30.6 EXCEPTIONS AND INTERRUPTS WITHIN SMM 


When the processor enters SMM, all hardware interrupts are disabled in the following manner: 


¢ The IF flag in the EFLAGS register is cleared, which inhibits maskable hardware interrupts from being 
generated. 


¢ The TF flag in the EFLAGS register is cleared, which disables single-step traps. 


° Debug register DR7 is cleared, which disables breakpoint traps. (This action prevents a debugger from acciden- 
tally breaking into an SMI handler if a debug breakpoint is set in normal address space that overlays code or 
data in SMRAM.) 


¢ NMI, SMI, and A20M interrupts are blocked by internal SMM logic. (See Section 30.8 for more information 
about how NMIs are handled in SMM.) 


Software-invoked interrupts and exceptions can still occur, and maskable hardware interrupts can be enabled by 
setting the IF flag. Intel recommends that SMM code be written in so that it does not invoke software interrupts 
(with the INT n, INTO, INT1, INT3, or BOUND instructions) or generate exceptions. 


If the SMI handler requires interrupt and exception handling, an SMM interrupt table and the necessary exception 
and interrupt handlers must be created and initialized from within SMM. Until the interrupt table is correctly initial- 
ized (using the LIDT instruction), exceptions and software interrupts will result in unpredictable processor 
behavior. 


The following restrictions apply when designing SMM interrupt and exception-handling facilities: 


° The interrupt table should be located at linear address 0 and must contain real-address mode style interrupt 
vectors (4 bytes containing CS and IP). 


* Due to the real-address mode style of base address formation, an interrupt or exception cannot transfer control 
to a segment with a base address of more that 20 bits. 


e An interrupt or exception cannot transfer control to a segment offset of more than 16 bits (64 KBytes). 


¢ When an exception or interrupt occurs, only the 16 least-significant bits of the return address (EIP) are pushed 
onto the stack. If the offset of the interrupted procedure is greater than 64 KBytes, it is not possible for the 
interrupt/exception handler to return control to that procedure. (One solution to this problem is for a handler 
to adjust the return address on the stack.) 


° The SMBASE relocation feature affects the way the processor will return from an interrupt or exception 
generated while the SMI handler is executing. For example, if the SMBASE is relocated to above 1 MByte, but 
the exception handlers are below 1 MByte, a normal return to the SMI handler is not possible. One solution is 
to provide the exception handler with a mechanism for calculating a return address above 1 MByte from the 16- 
bit return address on the stack, then use a 32-bit far call to return to the interrupted procedure. 


¢ If an SMI handler needs access to the debug trap facilities, it must ensure that an SMM accessible debug 
handler is available and save the current contents of debug registers DRO through DR3 (for later restoration). 
Debug registers DRO through DR3 and DR7 must then be initialized with the appropriate values. 


¢ If an SMI handler needs access to the single-step mechanism, it must ensure that an SMM accessible single- 
step handler is available, and then set the TF flag in the EFLAGS register. 


¢ If the SMI design requires the processor to respond to maskable hardware interrupts or software-generated 
interrupts while in SMM, it must ensure that SMM accessible interrupt handlers are available and then set the 
IF flag in the EFLAGS register (using the STI instruction). Software interrupts are not blocked upon entry to 
SMM, so they do not need to be enabled. 
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30.7. MANAGING SYNCHRONOUS AND ASYNCHRONOUS 
SYSTEM MANAGEMENT INTERRUPTS 


When coding for a multiprocessor system or a system with Intel HT Technology, it was not always possible for an 
SMI handler to distinguish between a synchronous SMI (triggered during an I/O instruction) and an asynchronous 
SMI. To facilitate the discrimination of these two events, incremental state information has been added to the SMM 


state save map. 
Processors that have an SMM revision ID of 30004H or higher have the incremental state information described 
below. 


30.7.1. 1/0 State Implementation 


Within the extended SMM state save map, a bit (IO_SMI) is provided that is set only when an SMI is either taken 
immediately after a successful I/O instruction or is taken after a successful iteration of a REP I/O instruction (the 
successful notion pertains to the processor point of view; not necessarily to the corresponding platform function). 
When set, the IO_SMI bit provides a strong indication that the corresponding SMI was synchronous. In this case, 
the SMM State Save Map also supplies the port address of the I/O operation. The IO_SMI bit and the I/O Port 
Address may be used in conjunction with the information logged by the platform to confirm that the SMI was 
indeed synchronous. 

The IO_SMI bit by itself is a strong indication, not a guarantee, that the SMI is synchronous. This is because an 
asynchronous SMI might coincidentally be taken after an I/O instruction. In such a case, the IO_SMI bit would still 
be set in the SMM state save map. 

Information characterizing the I/O instruction is saved in two locations in the SMM State Save Map (Table 30-5). 
The IO_SMI bit also serves as a valid bit for the rest of the I/O information fields. The contents of these I/O infor- 
mation fields are not defined when the IO_SMI bit is not set. 


Table 30-5. 1/0 Instruction Information in the SMM State Save Map 


State (SMM Rev. ID: 30004H or higher) Format 
31 16 15 8 7 4 3 1 0 
I/O State Field S © S S So 
SMRAM offset 7FA4 a © a & z 
a S 3 ra = 
o: S 
31 0 
I/O Memory Address Field I/O Memory Address 
SMRAM offset 7FAO 
When IO_SM is set, the other fields may be interpreted as follows: 
¢ 1/0 length: 
* 001 - Byte 
¢ 010 - Word 


¢ 100 - Dword 
¢ J/O instruction type (Table 30-6) 


Table 30-6. I/O Instruction Type Encodings 


Instruction Encoding 
IN Immediate 1001 
IN DX 0001 
OUT Immediate 1000 
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Table 30-6. I/O Instruction Type Encodings (Contd.) 


Instruction Encoding 
OUT DX 0000 
INS 0011 
OUTS 0010 
REP INS 0111 
REP OUTS 0110 


30.8 NMI HANDLING WHILE IN SMM 


NMI interrupts are blocked upon entry to the SMI handler. If an NMI request occurs during the SMI handler, it is 
latched and serviced after the processor exits SMM. Only one NMI request will be latched during the SMI handler. 
If an NMI request is pending when the processor executes the RSM instruction, the NMI is serviced before the next 
instruction of the interrupted code sequence. This assumes that NMIs were not blocked before the SMI occurred. If 
NMIs were blocked before the SMI occurred, they are blocked after execution of RSM. 


Although NMI requests are blocked when the processor enters SMM, they may be enabled through software by 
executing an IRET instruction. If the SMI handler requires the use of NMI interrupts, it should invoke a dummy 
interrupt service routine for the purpose of executing an IRET instruction. Once an IRET instruction is executed, 
NMI interrupt requests are serviced in the same “real mode” manner in which they are handled outside of SMM. 


Also, for the Pentium processor, exceptions that invoke a trap or fault handler will enable NMI interrupts from inside 
of SMM. This behavior is implementation specific for the Pentium processor and is not part of the IA-32 architec- 
ture. 


30.9 SMM REVISION IDENTIFIER 


The SMM revision identifier field is used to indicate the version of SMM and the SMM extensions that are supported 
by the processor (see Figure 30-2). The SMM revision identifier is written during SMM entry and can be examined 
in SMRAM space at offset 7EFCH. The lower word of the SMM revision identifier refers to the version of the base 
SMM architecture. 


Register Offset 

7EFCH 

31 18171615 0 
Reserved SMM Revision Identifier 


SMBASE Relocation 
1/O Instruction Restart 


Figure 30-2. SMM Revision Identifier 


The upper word of the SMM revision identifier refers to the extensions available. If the I/O instruction restart flag 
(bit 16) is set, the processor supports the I/O instruction restart (see Section 30.12); if the SMBASE relocation flag 
(bit 17) is set, SMRAM base address relocation is supported (see Section 30.11). 
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30.10 AUTO HALT RESTART 


If the processor is in a HALT state (due to the prior execution of a HLT instruction) when it receives an SMI, the 
processor records the fact in the auto HALT restart flag in the saved processor state (see Figure 30-3). (This flag is 
located at offset 7FO2H and bit 0 in the state save area of the SMRAM.) 


If the processor sets the auto HALT restart flag upon entering SMM (indicating that the SMI occurred when the 
processor was in the HALT state), the SMI handler has two options: 


¢ It can leave the auto HALT restart flag set, which instructs the RSM instruction to return program control to the 
HLT instruction. This option in effect causes the processor to re-enter the HALT state after handling the SMI. 
(This is the default operation. ) 


¢ It can clear the auto HALT restart flag, which instructs the RSM instruction to return program control to the 
instruction following the HLT instruction. 


15 10 


Register Offset 
Reserved 7F02H 


Auto HALT Restart — 


Figure 30-3. Auto HALT Restart Field 


These options are summarized in Table 30-7. If the processor was not in a HALT state when the SMI was received 
(the auto HALT restart flag is cleared), setting the flag to 1 will cause unpredictable behavior when the RSM instruc- 
tion is executed. 


Table 30-7. Auto HALT Restart Flag Values 


Value of Flag After | Value of Flag When Action of Processor When Exiting SMM 

Entry to SMM Exiting SMM 

0) 0) Returns to next instruction in interrupted program or task. 
0 1 Unpredictable. 

1 0 Returns to next instruction after HLT instruction. 

1 1 Returns to HALT state. 


If the HLT instruction is restarted, the processor will generate a memory access to fetch the HLT instruction (if it is 
not in the internal cache), and execute a HLT bus transaction. This behavior results in multiple HLT bus transactions 
for the same HLT instruction. 


30.10.1 Executing the HLT Instruction in SMM 


The HLT instruction should not be executed during SMM, unless interrupts have been enabled by setting the IF flag 
in the EFLAGS register. If the processor is halted in SMM, the only event that can remove the processor from this 
state is a maskable hardware interrupt or a hardware reset. 


30.11 SMBASE RELOCATION 


The default base address for the SMRAM is 30000H. This value is contained in an internal processor register called 
the SMBASE register. The operating system or executive can relocate the SMRAM by setting the SMBASE field in the 
saved state map (at offset 7EF8H) to a new value (see Figure 30-4). The RSM instruction reloads the internal 
SMBASE register with the value in the SMBASE field each time it exits SMM. All subsequent SMI requests will use 
the new SMBASE value to find the starting address for the SMI handler (at SMBASE + 8000H) and the SMRAM state 
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save area (from SMBASE + FEOQOH to SMBASE + FFFFH). (The processor resets the value in its internal SMBASE 
register to 30000H on a RESET, but does not change it on an INIT.) 


31 0 


Register Offset 


SMM Base 7EF8H 


Figure 30-4. SMBASE Relocation Field 


In multiple-processor systems, initialization software must adjust the SMBASE value for each processor so that the 
SMRAM state save areas for each processor do not overlap. (For Pentium and Intel486 processors, the SMBASE 
values must be aligned on a 32-KByte boundary or the processor will enter shutdown state during the execution of 
a RSM instruction.) 


If the SMBASE relocation flag in the SMM revision identifier field is set, it indicates the ability to relocate the 
SMBASE (see Section 30.9). 


30.12 I/O INSTRUCTION RESTART 


If the I/O instruction restart flag in the SMM revision identifier field is set (see Section 30.9), the I/O instruction 
restart mechanism is present on the processor. This mechanism allows an interrupted I/O instruction to be re- 
executed upon returning from SMM mode. For example, if an I/O instruction is used to access a powered-down I/O 
device, a chip set supporting this device can intercept the access and respond by asserting SMI#. This action 
invokes the SMI handler to power-up the device. Upon returning from the SMI handler, the I/O instruction restart 
mechanism can be used to re-execute the I/O instruction that caused the SMI. 


The I/O instruction restart field (at offset 7FOOH in the SMM state-save area, see Figure 30-5) controls I/O instruc- 
tion restart. When an RSM instruction is executed, if this field contains the value FFH, then the EIP register is modi- 
fied to point to the I/O instruction that received the SMI request. The processor will then automatically re-execute 
the I/O instruction that the SMI trapped. (The processor saves the necessary machine state to ensure that re- 
execution of the instruction is handled coherently. ) 


15 0 


F F Register Offset 
I/O Instruction Restart Field 
/O Instruction Restart Fie! 7FOOH 


Figure 30-5. 1/0 Instruction Restart Field 


If the I/O instruction restart field contains the value OOH when the RSM instruction is executed, then the processor 
begins program execution with the instruction following the I/O instruction. (When a repeat prefix is being used, 
the next instruction may be the next I/O instruction in the repeat loop.) Not re-executing the interrupted I/O 
instruction is the default behavior; the processor automatically initializes the I/O instruction restart field to OOH 
upon entering SMM. Table 30-8 summarizes the states of the I/O instruction restart field. 
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Table 30-8. 1/0 Instruction Restart Field Values 


Value of Flag After Value of Flag When Action of Processor When Exiting SMM 
Entry to SMM Exiting SMM 

OOH OOH Does not re-execute trapped I/O instruction. 
OOH FFH Re-executes trapped 1/0 instruction. 


The I/O instruction restart mechanism does not indicate the cause of the SMI. It is the responsibility of the SMI 
handler to examine the state of the processor to determine the cause of the SMI and to determine if an I/O instruc- 
tion was interrupted and should be restarted upon exiting SMM. If an SMI interrupt is signaled on a non-I/O instruc- 
tion boundary, setting the I/O instruction restart field to FFH prior to executing the RSM instruction will likely result 
in a program error. 


30.12.1 Back-to-Back SMI Interrupts When I/O Instruction Restart Is Being Used 


If an SMI interrupt is signaled while the processor is servicing an SMI interrupt that occurred on an I/O instruction 
boundary, the processor will service the new SMI request before restarting the originally interrupted I/O instruc- 
tion. If the I/O instruction restart field is set to FFH prior to returning from the second SMI handler, the EIP will point 
to an address different from the originally interrupted I/O instruction, which will likely lead to a program error. To 
avoid this situation, the SMI handler must be able to recognize the occurrence of back-to-back SMI interrupts when 
1/O instruction restart is being used and ensure that the handler sets the I/O instruction restart field to OOH prior 
to returning from the second invocation of the SMI handler. 


30.13. SMM MULTIPLE-PROCESSOR CONSIDERATIONS 


The following should be noted when designing multiple-processor systems: 
¢ Any processor in a multiprocessor system can respond to an SMM. 
¢ Each processor needs its own SMRAM space. This space can be in system memory or in a separate RAM. 


¢ The SMRAMs for different processors can be overlapped in the same memory space. The only stipulation is that 
each processor needs its own state save area and its own dynamic data storage area. (Also, for the Pentium 
and Intel486 processors, the SMBASE address must be located on a 32-KByte boundary.) Code and static data 
can be shared among processors. Overlapping SMRAM spaces can be done more efficiently with the P6 family 
processors because they do not require that the SMBASE address be on a 32-KByte boundary. 


¢ The SMI handler will need to initialize the SMBASE for each processor. 


* Processors can respond to local SMIs through their SMI# pins or to SMIs received through the APIC interface. 
The APIC interface can distribute SMIs to different processors. 


* Two or more processors can be executing in SMM at the same time. 


¢ When operating Pentium processors in dual processing (DP) mode, the SMIACT# pin is driven only by the MRM 
processor and should be sampled with ADS#. For additional details, see Chapter 14 of the Pentium Processor 
Family User’s Manual, Volume 1. 


SMM is not re-entrant, because the SMRAM State Save Map is fixed relative to the SMBASE. If there is a need to 
support two or more processors in SMM mode at the same time then each processor should have dedicated SMRAM 
spaces. This can be done by using the SMBASE Relocation feature (see Section 30.11). 


30.14 DEFAULT TREATMENT OF SMIS AND SMM WITH VMX OPERATION AND 
SMX OPERATION 


Under the default treatment, the interactions of SMIs and SMM with VMX operation are few. This section details 
those interactions. It also explains how this treatment affects SMX operation. 
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30.14.1 Default Treatment of SMI Delivery 


Ordinary SMI delivery saves processor state into SMRAM and then loads state based on architectural definitions. 
Under the default treatment, processors that support VMX operation perform SMI delivery as follows: 


enter SMM; 
save the following internal to the processor: 
CR4.VMXE 
an indication of whether the logical processor was in VMX operation (root or non-root) 
IF the logical processor is in VMX operation 
THEN 
save current VMCS pointer internal to the processor; 
leave VMX operation; 
save VMx-critical state defined below; 
Fl; 
IF the logical processor supports SMX operation 
THEN 
save internal to the logical processor an indication of whether the Intel® TXT private space is locked; 
IF the TXT private space is unlocked 
THEN lock the TXT private space; 
Fl; 
Fl; 
CR4.VMXE := 0; 
perform ordinary SMI delivery: 
save processor state in SMRAM; 
set processor state to standard SMM values;! 
invalidate linear mappings and combined mappings associated with VPID OOOOH (for all PCIDs); combined mappings for VPID OOOOH 
are invalidated for all EP4TA values (EP4TA is the value of bits 51:12 of EPTP; see Section 27.3); 


The pseudocode above makes reference to the saving of VMX-critical state. This state consists of the following: 
(1) SS.DPL (the current privilege level); (2) RFLAGS.VM2; (3) the state of blocking by STI and by MOV SS (see 
Table 23-3 in Section 23.4.2); (4) the state of virtual-NMI blocking (only if the processor is in VMX non-root oper- 
ation and the “virtual NMIs” VM-execution control is 1); and (5) an indication of whether an MTF VM exit is pending 
(see Section 24.5.2). These data may be saved internal to the processor or in the VMCS region of the current 
VMCS. Processors that do not support SMI recognition while there is blocking by STI or by MOV SS need not save 
the state of such blocking. 


If the logical processor supports the 1-setting of the “enable EPT” VM-execution control and the logical processor 
was in VMX non-root operation at the time of an SMI, it saves the value of that control into bit 0 of the 32-bit field 
at offset SMBASE + 8000H + 7EEOH (SMBASE + FEEOH; see Table 30-3).? If the logical processor was not in VMX 
non-root operation at the time of the SMI, it saves 0 into that bit. If the logical processor saves 1 into that bit (it 
was in VMX non-root operation and the “enable EPT” VM-execution control was 1), it saves the value of the EPT 
pointer (EPTP) into the 64-bit field at offset SMBASE + 8000H + 7ED8H (SMBASE + FED8H). 


Because SMI delivery causes a logical processor to leave VMX operation, all the controls associated with VMX non- 
root operation are disabled in SMM and thus cannot cause VM exits while the logical processor in SMM. 


1. This causes the logical processor to block INIT signals, NMIs, and SMIs. 

2. Section 30.14 and Section 30.15 use the notation RAX, RIP, RSP, RFLAGS, etc. for processor registers because most processors that 
support VMX operation also support Intel 64 architecture. For processors that do not support Intel 64 architecture, this notation 
refers to the 32-bit forms of these registers (EAX, EIP, ESP, EFLAGS, etc.). In a few places, notation such as EAX is used to refer spe- 
cifically to the lower 32 bits of the register. 

3. “Enable EPT” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution controls 
is O, SMI functions as the “enable EPT” V/M-execution control were 0. See Section 23.6.2. 
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30.14.2 Default Treatment of RSM 


Ordinary execution of RSM restores processor state from SMRAM. Under the default treatment, processors that 
support VMX operation perform RSM as follows: 


IF VMXE = 1 in CR4 image in SMRAM 
THEN fail and enter shutdown state; 
ELSE 
restore state normally from SMRAM; 
invalidate linear mappings and combined mappings associated with all VPIDs and all PCIDs; combined mappings are invalidated 
for all EP4TA values (EP4TA is the value of bits 51:12 of EPTP; see Section 27.3); 
IF the logical processor supports SMX operation andthe Intel® TXT private space was unlocked at the time of the last SMI (as 


saved) 
THEN unlock the TXT private space; 
Fl; 
CR4.VMXE := value stored internally; 
IF internal storage indicates that the logical processor 
had been in VMX operation (root or non-root) 
THEN 
enter VMX operation (root or non-root); 
restore VMX-critical state as defined in Section 30.14.1; 
set to their fixed values any bits in CRO and CR4 whose values must be fixed in VMX operation (see Section 22.8);! 
IF RFLAGS.VM = 0 AND (in VMX root operation OR the “unrestricted guest” VM-execution control is 0)2 
THEN 
CS.RPL := SS.DPL; 
SS.RPL := SS.DPL; 
Fl; 
restore current VMCS pointer; 
Fl; 
leave SMM; 
IF logical processor will be in VMX operation or in SMX operation after RSM 
THEN block AZ0M and leave A20M mode; 
Fl; 
Fl; 


RSM unblocks SMIs. It restores the state of blocking by NMI (see Table 23-3 in Section 23.4.2) as follows: 


¢ If the RSM is not to VMX non-root operation or if the “virtual NMIs” VM-execution control will be 0, the state of 
NMI blocking is restored normally. 


° If the RSM is to VMX non-root operation and the “virtual NMIs” VM-execution control will be 1, NMIs are not 
blocked after RSM. The state of virtual-NMI blocking is restored as part of VMX-critical state. 


INIT signals are blocked after RSM if and only if the logical processor will be in VMX root operation. 


If RSM returns a logical processor to VMX non-root operation, it re-establishes the controls associated with the 
current VMCS. If the “interrupt-window exiting” VM-execution control is 1, a VM exit occurs immediately after RSM 
if the enabling conditions apply. The same is true for the “NMI-window exiting” VM-execution control. Such 

VM exits occur with their normal priority. See Section 24.2. 


If an MTF VM exit was pending at the time of the previous SMI, an MTF VM exit is pending on the instruction 
boundary following execution of RSM. The following items detail the treatment of MTF VM exits that may be 
pending following RSM: 


1. If the RSM is to VMX non-root operation and both the “unrestricted guest” VM-execution control and bit 31 of the primary proces- 
sor-based VM-execution controls will be 1, CRO.PE and CRO.PG retain the values that were loaded from SMRAM regardless of what is 
reported in the capability MSR IA32_VMX_CRO_FIXEDO. 

2. “Unrestricted quest” is a secondary processor-based VM-execution control. If bit 31 of the primary processor-based VM-execution 
controls is 0, VM entry functions as if the “unrestricted quest” VM-execution control were O. See Section 23.6.2. 
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¢ System-management interrupts (SMIs), INIT signals, and higher priority events take priority over these MTF 
VM exits. These MTF VM exits take priority over debug-trap exceptions and lower priority events. 


° These MTF VM exits wake the logical processor if RSM caused the logical processor to enter the HLT state (see 
Section 30.10). They do not occur if the logical processor just entered the shutdown state. 


30.14.3 Protection of CR4.VMXE in SMM 


Under the default treatment, CR4.VMXE is treated as a reserved bit while a logical processor is in SMM. Any 
attempt by software running in SMM to set this bit causes a general-protection exception. In addition, software 
cannot use VMX instructions or enter VMX operation while in SMM. 


30.14.4 VWMXOFF and SMI Unblocking 


The VMXOFF instruction can be executed only with the default treatment (see Section 30.15.1) and only outside 
SMM. If SMIs are blocked when VMXOFF is executed, VMXOFF unblocks them unless 
IA32_SMM_MONITOR_CTLI[bit 2] is 1 (see Section 30.15.5 for details regarding this MSR).! Section 30.15.7 iden- 
tifies a case in which SMIs may be blocked when VMXOFF is executed. 


Not all processors allow this bit to be set to 1. Software should consult the VMX capability MSR IA32_VMX_MISC 
(see Appendix A.6) to determine whether this is allowed. 


30.15 DUAL-MONITOR TREATMENT OF SMis AND SMM 


Dual-monitor treatment is activated through the cooperation of the executive monitor (the VMM that operates 
outside of SMM to provide basic virtualization) and the SMM-transfer monitor (STM; the VMM that operates 
inside SMM—while in VMX operation—to support system-management functions). Control is transferred to the STM 
through VM exits; VM entries are used to return from SMM. 


The dual-monitor treatment may not be supported by all processors. Software should consult the VMX capability 
MSR IA32_VMX_BASIC (see Appendix A.1) to determine whether it is supported. 


30.15.1 Dual-Monitor Treatment Overview 


The dual-monitor treatment uses an executive monitor and an SMM-transfer monitor (STM). Transitions from the 
executive monitor or its guests to the STM are called SMM VM exits and are discussed in Section 30.15.2. SMM 
VM exits are caused by SMIs as well as executions of VMCALL in VMX root operation. The latter allow the executive 
monitor to call the STM for service. 


The STM runs in VMX root operation and uses VMX instructions to establish a VMCS and perform VM entries to its 
own guests. This is done all inside SMM (see Section 30.15.3). The STM returns from SMM, not by using the RSM 
instruction, but by using a VM entry that returns from SMM. Such VM entries are described in Section 30.15.4. 


Initially, there is no STM and the default treatment (Section 30.14) is used. The dual-monitor treatment is not used 
until it is enabled and activated. The steps to do this are described in Section 30.15.5 and Section 30.15.6. 


It is not possible to leave VMX operation under the dual-monitor treatment; VMXOFF will fail if executed. The dual- 
monitor treatment must be deactivated first. The STM deactivates dual-monitor treatment using a VM entry that 
returns from SMM with the “deactivate dual-monitor treatment” VM-entry control set to 1 (see Section 30.15.7). 


The executive monitor configures any VMCS that it uses for VM exits to the executive monitor. SMM VM exits, which 
transfer control to the STM, use a different VMCS. Under the dual-monitor treatment, each logical processor uses 
a separate VMCS called the SMM-transfer VMCS. When the dual-monitor treatment is active, the logical 
processor maintains another VMCS pointer called the SMM-transfer VMCS pointer. The SMM-transfer VMCS 
pointer is established when the dual-monitor treatment is activated. 


1. Setting IA32_SMM_MONITOR_CTL[bit 2] to 1 prevents VMXOFF from unblocking SMIs regardless of the value of the register’s valid 
bit (bit 0). 
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30.15.2 SMM VM Exits 
An SMM VM exit is a VM exit that begins outside SMM and that ends in SMM. 


Unlike other VM exits, SMM VM exits can begin in VMX root operation. SMM VM exits result from the arrival of an 
SMI outside SMM or from execution of VMCALL in VMX root operation outside SMM. Execution of VMCALL in VMX 
root operation causes an SMM VM exit only if the valid bit is set in the IA32_SMM_MONITOR_CTL MSR (see Section 
30.15.5). 


Execution of VMCALL in VMX root operation causes an SMM VM exit even under the default treatment. This SMM 
VM exit activates the dual-monitor treatment (see Section 30.15.6). 


Differences between SMM VM exits and other VM exits are detailed in Sections 30.15.2.1 through 30.15.2.5. 
Differences between SMM VM exits that activate the dual-monitor treatment and other SMM VM exits are described 
in Section 30.15.6. 


30.15.2.1 Architectural State Before a VM Exit 


System-management interrupts (SMIs) that cause SMM VM exits always do so directly. They do not save state to 
SMRAM as they do under the default treatment. 


30.15.2.2 Updating the Current-VMCS and Executive-VMCS Pointers 

SMM VM exits begin by performing the following steps: 

1. The executive-VMCS pointer field in the SMM-transfer VMCS is loaded as follows: 
— Ifthe SMM VM exit commenced in VMX non-root operation, it receives the current-VMCS pointer. 
— Ifthe SMM VM exit commenced in VMX root operation, it receives the VMXON pointer. 

2. The current-VMCS pointer is loaded with the value of the SMM-transfer VMCS pointer. 


The last step ensures that the current VMCS is the SMM-transfer VMCS. VM-exit information is recorded in that 
VMCS, and VM-entry control fields in that VMCS are updated. State is saved into the guest-state area of that VMCS. 
The VM-exit controls and host-state area of that VMCS determine how the VM exit operates. 


30.15.2.3 Recording VM-Exit Information 


SMM VM exits differ from other VM exit with regard to the way they record VM-exit information. The differences 
follow. 


e Exit reason. 


— Bits 15:0 of this field contain the basic exit reason. The field is loaded with the reason for the SMM VM exit: 
I/O SMI (an SMI arrived immediately after retirement of an I/O instruction), other SMI, or VMCALL. See 
Appendix C, “VMX Basic Exit Reasons”. 


— SMM VM exits are the only VM exits that may occur in VMX root operation. Because the SMM-transfer 
monitor may need to know whether it was invoked from VMX root or VMX non-root operation, this 
information is stored in bit 29 of the exit-reason field (see Table 23-16 in Section 23.9.1). The bit is set by 
SMM VM exits from VMX root operation. 


— Ifthe SMM VM exit occurred in VMX non-root operation and an MTF VM exit was pending, bit 28 of the exit- 
reason field is set; otherwise, it is cleared. 


— Bits 27:16 and bits 31:30 are cleared. 


° Exit qualification. For an SMM VM exit due an SMI that arrives immediately after the retirement of an I/O 
instruction, the exit qualification contains information about the I/O instruction that retired immediately before 
the SMI. It has the format given in Table 30-9. 


* Guest linear address. This field is used for VM exits due to SMIs that arrive immediately after the retirement 
of an INS or OUTS instruction for which the relevant segment (ES for INS; DS for OUTS unless overridden by 
an instruction prefix) is usable. The field receives the value of the linear address generated by ES:(E)DI (for 
INS) or segment:(E)SI (for OUTS; the default segment is DS but can be overridden by a segment override 
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Table 30-9. Exit Qualification for SMls That Arrive Immediately After the Retirement of an I/O Instruction 


Bit Position(s) Contents 

2:0 Size of access: 
0 = 1-byte 
1 = 2-byte 
3 = 4-byte 


Other values not used. 


3 Direction of the attempted access (0 = OUT, 1 = IN) 

4 String instruction (0 = not string; 1 = string) 

5 REP prefixed (O = not REP; 1 = REP) 

6 Operand encoding (0 = DX, 1 = immediate) 

15:7 Reserved (cleared to 0) 

31:16 Port number (as specified in the I/O instruction) 

63:32 Reserved (cleared to 0). These bits exist only on processors 


that support Intel 64 architecture. 


prefix) at the time the instruction started. If the relevant segment is not usable, the value is undefined. On 
processors that support Intel 64 architecture, bits 63:32 are clear if the logical processor was not in 64-bit 
mode before the VM exit. 


¢ 1/0 RCX, I/O RSI, I/O RDI, and I/O RIP. For an SMM VM exit due an SMI that arrives immediately after 
the retirement of an I/O instruction, these fields receive the values that were in RCX, RSI, RDI, and RIP, respec- 
tively, before the I/O instruction executed. Thus, the value saved for I/O RIP addresses the I/O instruction. 


30.15.2.4 Saving Guest State 
SMM VM exits save the contents of the SMBASE register into the corresponding field in the guest-state area. 


The value of the VMX-preemption timer is saved into the corresponding field in the guest-state area if the “save 
VMX-preemption timer value” VM-exit control is 1. That field becomes undefined if, in addition, either the SMM 
VM exit is from VMX root operation or the SMM VM exit is from VMX non-root operation and the “activate VMX- 
preemption timer” VM-execution control is 0. 


30.15.2.5 Updating State 


If an SMM VM exit is from VMX non-root operation and the “Intel PT uses guest physical addresses” VM-execution 
control is 1, the IA32_RTIT_CTL MSR is cleared to 00000000_00000000H.! This is done even if the “clear 
IA32_RTIT_CTL” VM-exit control is 0. 


SMM VM exits affect the non-register state of a logical processor as follows: 


¢ SMM VM exits cause non-maskable interrupts (NMIs) to be blocked; they may be unblocked through execution 
of IRET or through a VM entry (depending on the value loaded for the interruptibility state and the setting of 
the “virtual NMIs” VM-execution control). 


¢ SMM VM exits cause SMIs to be blocked; they may be unblocked by a VM entry that returns from SMM (see 
Section 30.15.4). 


1. In this situation, the value of this MSR was saved earlier into the quest-state area. All VM exits save this MSR if the 1-setting of the 
“load IA32_RTIT_CTL” VM-entry control is supported (see Section 26.3.1), which must be the case if the “Intel PT uses guest physi- 
cal addresses” VM-execution control is 1 (see Section 25.2.1.1). 
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SMM VM exits invalidate linear mappings and combined mappings associated with VPID OOOOH for all PCIDs. 
Combined mappings for VPID OOOOH are invalidated for all EP4TA values (EP4TA is the value of bits 51:12 of EPTP; 
see Section 27.3). (Ordinary VM exits are not required to perform such invalidation if the “enable VPID” VM-execu- 
tion control is 1; see Section 26.5.5.) 


30.15.3 Operation of the SMM-Transfer Monitor 


Once invoked, the SMM-transfer monitor (STM) is in VMX root operation and can use VMX instructions to configure 
VMCSs and to cause VM entries to virtual machines supported by those structures. As noted in Section 30.15.1, the 
VMXOFF instruction cannot be used under the dual-monitor treatment and thus cannot be used by the STM. 


The RSM instruction also cannot be used under the dual-monitor treatment. As noted in Section 24.1.3, it causes a 
VM exit if executed in SMM in VMX non-root operation. If executed in VMX root operation, it causes an invalid- 
opcode exception. The STM uses VM entries to return from SMM (see Section 30.15.4). 


30.15.4 VM Entries that Return from SMM 


The SMM-transfer monitor (STM) returns from SMM using a VM entry with the “entry to SMM” VM-entry control 
clear. VM entries that return from SMM reverse the effects of an SMM VM exit (see Section 30.15.2). 


VM entries that return from SMM may differ from other VM entries in that they do not necessarily enter VMX non- 
root operation. If the executive-VMCS pointer field in the current VMCS contains the VMXON pointer, the logical 
processor remains in VMX root operation after VM entry. 


For differences between VM entries that return from SMM and other VM entries see Sections 30.15.4.1 through 
30.15.4.10. 


30.15.4.1 Checks on the Executive-VMCS Pointer Field 


VM entries that return from SMM perform the following checks on the executive-VMCS pointer field in the current 
VMCS: 


¢ Bits 11:0 must be 0. 
* The pointer must not set any bits beyond the processor’s physical-address width.!/2 


¢ The 32 bits located in memory referenced by the physical address in the pointer must contain the processor's 
VMCS revision identifier (see Section 23.2). 


The checks above are performed before the checks described in Section 30.15.4.2 and before any of the following 
checks: 


° ‘If the “deactivate dual-monitor treatment” VM-entry control is 0 and the executive-VMCS pointer field does not 
contain the VMXON pointer, the launch state of the executive VMCS (the VMCS referenced by the executive- 
VMCS pointer field) must be launched (see Section 23.11.3). 


° If the “deactivate dual-monitor treatment” VM-entry control is 1, the executive-VMCS pointer field must 
contain the VMXON pointer (see Section 30.15.7).? 


30.15.4.2 Checks on VM-Execution Control Fields 


VM entries that return from SMM differ from other VM entries with regard to the checks performed on the VM- 
execution control fields specified in Section 25.2.1.1. They do not apply the checks to the current VMCS. Instead, 
VM-entry behavior depends on whether the executive-VMCS pointer field contains the VMXON pointer: 


1. Software can determine a processor's physical-address width by executing CPUID with BOOO0008H in EAX. The physical-address 
width is returned in bits 7:0 of EAX. 


2. If 1A32_VMX_BASIC[48] is read as 1, this pointer must not set any bits in the range 63:32; see Appendix A.1. 


3. The STM can determine the VMXON pointer by reading the executive-VMCS pointer field in the current VMCS after the SMM VM exit 
that activates the dual-monitor treatment. 


30-22 Vol. 3C 


SYSTEM MANAGEMENT MODE 


¢ = If the executive-VMCS pointer field contains the VMXON pointer (the VM entry remains in VMX root operation), 
the checks are not performed at all. 


¢ If the executive-VMCS pointer field does not contain the VMXON pointer (the VM entry enters VMX non-root 
operation), the checks are performed on the VM-execution control fields in the executive VMCS (the VMCS 
referenced by the executive-VMCS pointer field in the current VMCS). These checks are performed after 
checking the executive-VMCS pointer field itself (for proper alignment). 


Other VM entries ensure that, if “activate VMX-preemption timer” VM-execution control is 0, the “save VMX- 
preemption timer value” VM-exit control is also 0. This check is not performed by VM entries that return from SMM. 


30.15.4.3 Checks on VM-Entry Control Fields 


VM entries that return from SMM differ from other VM entries with regard to the checks performed on the VM-entry 
control fields specified in Section 25.2.1.3. 


Specifically, if the executive-VMCS pointer field contains the VMXON pointer (the VM entry remains in VMX root 
operation), the VM-entry interruption-information field must not indicate injection of a pending MTF VM exit (see 
Section 25.6.2). Specifically, the following cannot all be true for that field: 


¢ the valid bit (bit 31) is 1 
¢ the interruption type (bits 10:8) is 7 (other event); and 
¢ the vector (bits 7:0) is 0 (pending MTF VM exit). 


30.15.4.4 Checks on the Guest State Area 


Section 25.3.1 specifies checks performed on fields in the guest-state area of the VMCS. Some of these checks are 
conditioned on the settings of certain VM-execution controls (e.g., “virtual NMIs” or “unrestricted guest”). 

VM entries that return from SMM modify these checks based on whether the executive-VMCS pointer field contains 
the VMXON pointer:! 


¢ If the executive-VMCS pointer field contains the VMXON pointer (the VM entry remains in VMX root operation), 
the checks are performed as all relevant VM-execution controls were 0. (As a result, some checks may not be 
performed at all.) 


e If the executive-VMCS pointer field does not contain the VMXON pointer (the VM entry enters VMX non-root 
operation), this check is performed based on the settings of the VM-execution controls in the executive VMCS 
(the VMCS referenced by the executive-VMCS pointer field in the current VMCS). 


For VM entries that return from SMM, the activity-state field must not indicate the wait-for-SIPI state if the execu- 
tive-VMCS pointer field contains the VMXON pointer (the VM entry is to VMX root operation). 


30.15.4.5 Loading Guest State 
VM entries that return from SMM load the SMBASE register from the SMBASE field. 


VM entries that return from SMM invalidate linear mappings and combined mappings associated with all VPIDs. 
Combined mappings are invalidated for all EP4TA values (EP4TA is the value of bits 51:12 of EPTP; see Section 
27.3). (Ordinary VM entries are required to perform such invalidation only for VPID OOOOH and are not required to 
do even that if the “enable VPID” VM-execution control is 1; see Section 25.3.2.5.) 


30.15.4.6 VMX-Preemption Timer 


A VM entry that returns from SMM activates the VMX-preemption timer only if the executive-VMCS pointer field 
does not contain the VMXON pointer (the VM entry enters VMX non-root operation) and the “activate VMX-preemp- 
tion timer” VM-execution control is 1 in the executive VMCS (the VMCS referenced by the executive-VMCS pointer 
field). In this case, VM entry starts the VMX-preemption timer with the value in the VMX-preemption timer-value 
field in the current VMCS. 


1. The STM can determine the VMXON pointer by reading the executive-VMCS pointer field in the current VMCS after the SMM VM exit 
that activates the dual-monitor treatment. 
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30.15.4.7 Updating the Current-VMCS and SMM-Transfer VMCS Pointers 


Successful VM entries (returning from SMM) load the SMM-transfer VMCS pointer with the current-VMCS pointer. 
Following this, they load the current-VMCS pointer from a field in the current VMCS: 


° If the executive-VMCS pointer field contains the VMXON pointer (the VM entry remains in VMX root operation), 
the current-VMCS pointer is loaded from the VMCS-link pointer field. 


° If the executive-VMCS pointer field does not contain the VMXON pointer (the VM entry enters VMX non-root 
operation), the current-VMCS pointer is loaded with the value of the executive-VMCS pointer field. 


If the VM entry successfully enters VMX non-root operation, the VM-execution controls in effect after the VM entry 
are those from the new current VMCS. This includes any structures external to the VMCS referenced by VM-execu- 
tion control fields. 


The updating of these VMCS pointers occurs before event injection. Event injection is determined, however, by the 
VM-entry control fields in the VMCS that was current when the VM entry commenced. 


30.15.4.8 VM Exits Induced by VM Entry 


Section 25.6.1.2 describes how the event-delivery process invoked by event injection may lead to a VM exit. 
Section 25.7.3 to Section 25.7.7 describe other situations that may cause a VM exit to occur immediately after a 
VM entry. 


Whether these VM exits occur is determined by the VM-execution control fields in the current VMCS. For VM entries 
that return from SMM, they can occur only if the executive-VMCS pointer field does not contain the VMXON pointer 
(the VM entry enters VMX non-root operation). 


In this case, determination is based on the VM-execution control fields in the VMCS that is current after the 

VM entry. This is the VMCS referenced by the value of the executive-VMCS pointer field at the time of the VM entry 
(see Section 30.15.4.7). This VMCS also controls the delivery of such VM exits. Thus, VM exits induced by a 

VM entry returning from SMM are to the executive monitor and not to the STM. 


30.15.4.9 SMI Blocking 


VM entries that return from SMM determine the blocking of system-management interrupts (SMIs) as follows: 


¢ If the “deactivate dual-monitor treatment” VM-entry control is 0, SMIs are blocked after VM entry if and only if 
the bit 2 in the interruptibility-state field is 1. 


e If the “deactivate dual-monitor treatment” VM-entry control is 1, the blocking of SMIs depends on whether the 
logical processor is in SMX operation:! 


— Ifthe logical processor is in SMX operation, SMIs are blocked after VM entry. 
— Ifthe logical processor is outside SMX operation, SMIs are unblocked after VM entry. 


VM entries that return from SMM and that do not deactivate the dual-monitor treatment may leave SMIs blocked. 
This feature exists to allow the STM to invoke functionality outside of SMM without unblocking SMIs. 


30.15.4.10 Failures of VM Entries That Return from SMM 


Section 25.8 describes the treatment of VM entries that fail during or after loading guest state. Such failures record 
information in the VM-exit information fields and load processor state as would be done on a VM exit. The VMCS 
used is the one that was current before the VM entry commenced. Control is thus transferred to the STM and the 
logical processor remains in SMM. 


1. A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. A logi- 
cal processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the last 
execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference,” in the Intel® 64 and IA-32 Architectures Soft- 
ware Developer's Manual, Volume 2D. 
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30.15.5 Enabling the Dual-Monitor Treatment 


Code and data for the SMM-transfer monitor (STM) reside in a region of SMRAM called the monitor segment 
(MSEG). Code running in SMM determines the location of MSEG and establishes its content. This code is also 
responsible for enabling the dual-monitor treatment. 


SMM code enables the dual-monitor treatment and specifies the location of MSEG by writing to the 
IA32_SMM_MONITOR_CTL MSR (index 9BH). The MSR has the following format: 


Bit 0 is the register’s valid bit. The STM may be invoked using VMCALL only if this bit is 1. Because VMCALL is 
used to activate the dual-monitor treatment (see Section 30.15.6), the dual-monitor treatment cannot be 
activated if the bit is 0. This bit is cleared when the logical processor is reset. 


Bit 1 is reserved. 


Bit 2 determines whether executions of VMXOFF unblock SMIs under the default treatment of SMIs and SMM. 
Executions of VMXOFF unblock SMIs unless bit 2 is 1 (the value of bit 0 is irrelevant). See Section 30.14.4. 


Certain leaf functions of the GETSEC instruction clear this bit (see Chapter 6, “Safer Mode Extensions 
Reference,” in Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2D). 


Bits 11:3 are reserved. 


Bits 31:12 contain a value that, when shifted left 12 bits, is the physical address of MSEG (the MSEG base 
address). 


Bits 63:32 are reserved. 


The following items detail use of this MSR: 


The IA32_SMM_MONITOR_CTL MSR is supported only on processors that support the dual-monitor treatment.! 
On other processors, accesses to the MSR using RDMSR or WRMSR generate a general-protection fault 
(#GP(0)). 


A write to the IA32_SMM_MONITOR_CTL MSR using WRMSR generates a general-protection fault (#GP(0)) if 
executed outside of SMM or if an attempt is made to set any reserved bit. An attempt to write to the 
IA32_SMM_MONITOR_CTL MSR fails if made as part of a VM exit that does not end in SMM or part of a 

VM entry that does not begin in SMM. 


Reads from the IA32_SMM_MONITOR_CTL MSR using RDMSR are allowed any time RDMSR is allowed. The 
MSR may be read as part of any VM exit. 


The dual-monitor treatment can be activated only if the valid bit in the MSR is set to 1. 


The 32 bytes located at the MSEG base address are called the MSEG header. The format of the MSEG header is 
given in Table 30-10 (each field is 32 bits). 


Table 30-10. Format of MSEG Header 


Byte Offset Field 

0 MSEG-header revision identifier 
4 SMM-transfer monitor features 
8 GDTR limit 

12 GDTR base offset 

16 CS selector 

20 EIP offset 

24 ESP offset 

28 CR3 offset 


1. 


Software should consult the VMX capability MSR I[A32_VMX_BASIC (see Appendix A.1) to determine whether the dual-monitor 
treatment is supported. 
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To ensure proper behavior in VMX operation, software should maintain the MSEG header in writeback cacheable 
memory. Future implementations may allow or require a different memory type.! Software should consult the VMX 
capability MSR IA32_VMX_BASIC (see Appendix A.1). 


SMM code should enable the dual-monitor treatment (by setting the valid bit in IA32_SMM_MONITOR_CTL MSR) 
only after establishing the content of the MSEG header as follows: 


¢ Bytes 3:0 contain the MSEG revision identifier. Different processors may use different MSEG revision identi- 
fiers. These identifiers enable software to avoid using an MSEG header formatted for one processor ona 
processor that uses a different format. Software can discover the MSEG revision identifier that a processor uses 
by reading the VMX capability MSR IA32_VMX_MISC (see Appendix A.6). 


¢ Bytes 7:4 contain the SMM-transfer monitor features field. Bits 31:1 of this field are reserved and must be 
zero. Bit O of the field is the IA-32e mode SMM feature bit. It indicates whether the logical processor will be 
in IA-32e mode after the STM is activated (see Section 30.15.6). 


¢ Bytes 31:8 contain fields that determine how processor state is loaded when the STM is activated (see Section 
30.15.6.5). SMM code should establish these fields so that activating of the STM invokes the STM’s initialization 
code. 


30.15.6 Activating the Dual-Monitor Treatment 


The dual-monitor treatment may be enabled by SMM code as described in Section 30.15.5. The dual-monitor treat- 
ment is activated only if it is enabled and only by the executive monitor. The executive monitor activates the dual- 
monitor treatment by executing VMCALL in VMX root operation. 


When VMCALL activates the dual-monitor treatment, it causes an SMM VM exit. Differences between this SMM 
VM exit and other SMM VM exits are discussed in Sections 30.15.6.1 through 30.15.6.6. See also “VMCALL—Call to 
VM Monitor” in Chapter 29. 


30.15.6.1 Initial Checks 


An execution of VMCALL attempts to activate the dual-monitor treatment if (1) the processor supports the dual- 
monitor treatment; (2) the logical processor is in VMX root operation; (3) the logical processor is outside SMM and 
the valid bit is set in the IA32_SMM_MONITOR_CTL MSR; (4) the logical processor is not in virtual-8086 mode and 
not in compatibility mode; (5) CPL = 0; and (6) the dual-monitor treatment is not active. 


Such an execution of VMCALL begins with some initial checks. These checks are performed before updating the 
current-VMCS pointer and the executive-VMCS pointer field (see Section 30.15.2.2). 


The VMCS that manages SMM VM exit caused by this VMCALL is the current VMCS established by the executive 
monitor. The VMCALL performs the following checks on the current VMCS in the order indicated: 


1. There must be a current VMCS pointer. 
2. The launch state of the current VMCS must be clear. 


3. Reserved bits in the VM-exit controls in the current VMCS must be set properly. Software may consult the VMX 
capability MSR IA32_VMX_EXIT_CTLS to determine the proper settings (see Appendix A.4). 


If any of these checks fail, subsequent checks are skipped and VMCALL fails. If all these checks succeed, the logical 
processor uses the IA32_SMM_MONITOR_CTL MSR to determine the base address of MSEG. The following checks 
are performed in the order indicated: 


1. The logical processor reads the 32 bits at the base of MSEG and compares them to the processor’s MSEG 
revision identifier. 


1. Alternatively, software may map the MSEG header with the UC memory type; this may be necessary, depending on how memory is 
organized. Doing so is strongly discouraged unless necessary as it will cause the performance of transitions using those structures 
to suffer significantly. In addition, the processor will continue to use the memory type reported in the VMX capability MSR 
IA32_VMX_BASIC with exceptions noted in Appendix A.1. 

2. Software should consult the VMX capability MSR IA32_VMX_BASIC (see Appendix A.1) to determine whether the dual-monitor 
treatment is supported. 
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2. The logical processor reads the SMM-transfer monitor features field: 


— Bit 0 of the field is the IA-32e mode SMM feature bit, and it indicates whether the logical processor will be 
in IA-32e mode after the SMM-transfer monitor (STM) is activated. 


¢ If the VMCALL is executed on a processor that does not support Intel 64 architecture, the IA-32e mode 
SMM feature bit must be 0. 


¢ If the VMCALL is executed in 64-bit mode, the IA-32e mode SMM feature bit must be 1. 
— Bits 31:1 of this field are currently reserved and must be zero. 
If any of these checks fail, subsequent checks are skipped and the VMCALL fails. 


30.15.6.2 Updating the Current-VMCS and Executive-VMCS Pointers 


Before performing the steps in Section 30.15.2.2, SMM VM exits that activate the dual-monitor treatment begin by 
loading the SMM-transfer VMCS pointer with the value of the current-VMCS pointer. 


30.15.6.3 Saving Guest State 


As noted in Section 30.15.2.4, SMM VM exits save the contents of the SMBASE register into the corresponding field 
in the guest-state area. While this is true also for SMM VM exits that activate the dual-monitor treatment, the 
VMCS used for those VM exits exists outside SMRAM. 


The SMM-transfer monitor (STM) can also discover the current value of the SMBASE register by using the RDMSR 
instruction to read the IA32_SMBASE MSR (MSR address 9EH). The following items detail use of this MSR: 


¢ The MSR is supported only if IA32_VMX_MISC[15] = 1 (see Appendix A.6). 


° Awrite to the IA32_SMBASE MSR using WRMSR generates a general-protection fault (#GP(0)). An attempt to 
write to the IA32_SMBASE MSR fails if made as part of a VM exit or part of a VM entry. 


° Aread from the IA32_SMBASE MSR using RDMSR generates a general-protection fault (#GP(0)) if executed 
outside of SMM. An attempt to read from the IA32_SMBASE MSR fails if made as part of a VM exit that does not 
end in SMM. 


30.15.6.4 Saving MSRs 


The VM-exit MSR-store area is not used by SMM VM exits that activate the dual-monitor treatment. No MSRs are 
saved into that area. 


30.15.6.5 Loading Host State 


The VMCS that is current during an SMM VM exit that activates the dual-monitor treatment was established by the 
executive monitor. It does not contain the VM-exit controls and host state required to initialize the STM. For this 
reason, such SMM VM exits do not load processor state as described in Section 26.5. Instead, state is set to fixed 
values or loaded based on the content of the MSEG header (see Table 30-10): 


* CRO is set to as follows: 
— PG, NE, ET, MP, and PE are all set to 1. 
— CD and NW are left unchanged. 
— All other bits are cleared to 0. 
°* CR3 is set as follows: 
— Bits 63:32 are cleared on processors that support IA-32e mode. 


— Bits 31:12 are set to bits 31:12 of the sum of the MSEG base address and the CR3-offset field in the MSEG 
header. 


— Bits 11:5 and bits 2:0 are cleared (the corresponding bits in the CR3-offset field in the MSEG header are 
ignored). 
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— Bits 4:3 are set to bits 4:3 of the CR3-offset field in the MSEG header. 
° CR¢4 is set as follows: 

— MCE, PGE, CET, and PCIDE are cleared. 

— PAE is set to the value of the IA-32e mode SMM feature bit. 


— Ifthe IA-32e mode SMM feature bit is clear, PSE is set to 1 if supported by the processor; if the bit is set, 
PSE is cleared. 


— All other bits are unchanged. 
° DR7 is set to 400H. 
¢ The IA32_DEBUGCTL MSR is cleared to 0O000000_O00000000H. 
° The registers CS, SS, DS, ES, FS, and GS are loaded as follows: 
— All registers are usable. 


— CS.selector is loaded from the corresponding field in the MSEG header (the high 16 bits are ignored), with 
bits 2:0 cleared to O. If the result is OOOOH, CS.selector is set to OOO8H. 


— The selectors for SS, DS, ES, FS, and GS are set to CS.selector+0008H. If the result is OOOOH (if the CS 
selector was FFF8H), these selectors are instead set to OOO8H. 


— The base addresses of all registers are cleared to zero. 
— The segment limits for all registers are set to FFFFFFFFH. 
— The AR bytes for the registers are set as follows: 
¢ CS.Type is set to 11 (execute/read, accessed, non-conforming code segment). 
¢ For SS, DS, ES, FS, and GS, the Type is set to 3 (read/write, accessed, expand-up data segment). 
¢ The S bits for all registers are set to 1. 
¢ The DPL for each register is set to 0. 
¢ The P bits for all registers are set to 1. 


* On processors that support Intel 64 architecture, CS.Lis loaded with the value of the IA-32e mode SMM 
feature bit. 


¢ CS.D is loaded with the inverse of the value of the IA-32e mode SMM feature bit. 
¢ For each of SS, DS, ES, FS, and GS, the D/B bit is set to 1. 
¢ The G bits for all registers are set to 1. 


¢ LDTR is unusable. The LDTR selector is cleared to OOOOH, and the register is otherwise undefined (although the 
base address is always canonical) 


° GDTR.base is set to the sum of the MSEG base address and the GDTR base-offset field in the MSEG header 
(bits 63:32 are always cleared on processors that support IA-32e mode). GDTR. limit is set to the corresponding 
field in the MSEG header (the high 16 bits are ignored). 


¢ IDTR.base is unchanged. IDTR.|limit is cleared to OOOOH. 


° RIP is set to the sum of the MSEG base address and the value of the RIP-offset field in the MSEG header 
(bits 63:32 are always cleared on logical processors that support IA-32e mode). 


® RSP is set to the sum of the MSEG base address and the value of the RSP-offset field in the MSEG header 
(bits 63:32 are always cleared on logical processor that supports IA-32e mode). 


® RFLAGS is cleared, except bit 1, which is always set. 
¢ The logical processor is left in the active state. 
¢ Event blocking after the SMM VM exit is as follows: 
— There is no blocking by STI or by MOV SS. 
— There is blocking by non-maskable interrupts (NMIs) and by SMIs. 
¢ There are no pending debug exceptions after the SMM VM exit. 
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¢ For processors that support IA-32e mode, the IA32_EFER MSR is modified so that LME and LMA both contain 
the value of the IA-32e mode SMM feature bit. 


If any of CR3[63:5], CR4.PAE, CR4.PSE, or IA32_EFER.LMA is changing, the TLBs are updated so that, after 

VM exit, the logical processor does not use translations that were cached before the transition. This is not neces- 
sary for changes that would not affect paging due to the settings of other bits (for example, changes to CR4.PSE if 
IA32_EFER.LMA was 1 before and after the transition). 


30.15.6.6 Loading MSRs 


The VM-exit MSR-load area is not used by SMM VM exits that activate the dual-monitor treatment. No MSRs are 
loaded from that area. 


30.15.7 Deactivating the Dual-Monitor Treatment 


The SMM-transfer monitor may deactivate the dual-monitor treatment and return the processor to default treat- 
ment of SMIs and SMM (see Section 30.14). It does this by executing a VM entry with the “deactivate dual-monitor 
treatment” VM-entry control set to 1. 


As noted in Section 25.2.1.3 and Section 30.15.4.1, an attempt to deactivate the dual-monitor treatment fails in 
the following situations: (1) the processor is not in SMM; (2) the “entry to SMM” VM-entry control is 1; or (3) the 
executive-VMCS pointer does not contain the VMXON pointer (the VM entry is to VMX non-root operation). 


As noted in Section 30.15.4.9, VM entries that deactivate the dual-monitor treatment ignore the SMI bit in the 
interruptibility-state field of the guest-state area. Instead, the blocking of SMIs following such a VM entry depends 
on whether the logical processor is in SMX operation:! 


¢ If the logical processor is in SMX operation, SMIs are blocked after VM entry. SMIs may later be unblocked by 
the VMXOFF instruction (see Section 30.14.4) or by certain leaf functions of the GETSEC instruction (see 
Chapter 6, “Safer Mode Extensions Reference,” in Intel® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 2D). 


¢ If the logical processor is outside SMX operation, SMIs are unblocked after VM entry. 


30.16 SMI AND PROCESSOR EXTENDED STATE MANAGEMENT 


On processors that support processor extended states using XSAVE/XRSTOR (see Chapter 13, “Managing State 
Using the XSAVE Feature Set” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1), 
the processor does not save any XSAVE/XRSTOR related state on an SMI. It is the responsibility of the SMI handler 
code to properly preserve the state information (including CR4.0SXSAVE, XCRO, and possibly processor extended 
states using XSAVE/XRSTOR). Therefore, the SMI handler must follow the rules described in Chapter 13, 
“Managing State Using the XSAVE Feature Set” of the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 1. 


30.17 MODEL-SPECIFIC SYSTEM MANAGEMENT ENHANCEMENT 


This section describes enhancement of system management features that apply only to the 4th generation Intel 
Core processors. These features are model-specific. BIOS and SMM handler must use CPUID to enumerate 
DisplayFamily_DisplayModel signature when programming with these interfaces. 


1. A logical processor is in SMX operation if GETSEC[SEXIT] has not been executed since the last execution of GETSEC[SENTER]. A logi- 
cal processor is outside SMX operation if GETSEC[SENTER] has not been executed or if GETSEC[SEXIT] was executed after the last 
execution of GETSEC[SENTER]. See Chapter 6, “Safer Mode Extensions Reference,” in Intel® 64 and IA-32 Architectures Software 
Developer's Manual, Volume 2B. 
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30.17.1 SMM Handler Code Access Control 


The BIOS may choose to restrict the address ranges of code that SMM handler executes. When SMM handler code 
execution check is enabled, an attempt by the SMM handler to execute outside the ranges specified by SMRR (see 
Section 30.4.2.1) will cause the assertion of an unrecoverable machine check exception (MCE). 


The interface to enable SMM handler code access check resides in a per-package scope model-specific register 
MSR_SMM_FEATURE_CONTROL at address 4E0H. An attempt to access MSR_SMM_FEATURE_CONTROL outside of 
SMM will cause a #GP. Writes to MSR_SMM_FEATURE_CONTROL is further protected by configuration interface of 
MSR_SMM_MCA_CAP at address 17DH. 


Details of the interface of MSR_SMM_FEATURE_CONTROL and MSR_SMM_MCA_CAP are described in Table 2-29 in 
Chapter 2, “Model-Specific Registers (MSRs)” of the Inte/® 64 and IA-32 Architectures Software Developer’s 
Manual, Volume 4. 


30.17.2 SMI Delivery Delay Reporting 


Entry into the system management mode occurs at instruction boundary. In situations where a logical processor is 
executing an instruction involving a long flow of internal operations, servicing an SMI by that logical processor will 
be delayed. Delayed servicing of SMI of each logical processor due to executing long flows of internal operation in 
a physical processor can be queried via a package-scope register MSR_SMM_DELAYED at address 4E2H. 


The interface to enable reporting of SMI delivery delay due to long internal flows resides in a per-package scope 
model-specific register MSR_SMM_DELAYED. An attempt to access MSR_SMM_DELAYED outside of SMM will cause 
a #GP. Availability to MSR_SMM_DELAYED is protected by configuration interface of MSR_SMM_MCA_CAP at 
address 17DH. 


Details of the interface of MSR_SMM_DELAYED and MSR_SMM_MCA_CAP are described in Table 2-29 in Chapter 2, 
“Model-Specific Registers (MSRs)” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 
4. 


30.17.3 Blocked SMI Reporting 


A logical processor may have entered into a state and blocked from servicing other interrupts (including SMI). 
Logical processors in a physical processor that are blocked in serving SMI can be queried in a package-scope 
register MSR_SMM_BLOCKED at address 4E3H. An attempt to access MSR_SMM_BLOCKED outside of SMM will 
cause a #GP. 


Details of the interface of MSR_SMM_BLOCKED is described in Table 2-29 in Chapter 2, “Model-Specific Registers 
(MSRs)” of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4. 
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CHAPTER 31 
INTEL® PROCESSOR TRACE 


31.1 OVERVIEW 


Intel® Processor Trace (Intel PT) is an extension of Intel® Architecture that captures information about software 
execution using dedicated hardware facilities that cause only minimal performance perturbation to the software 
being traced. This information is collected in data packets. The initial implementations of Intel PT offer control 
flow tracing, which generates a variety of packets to be processed by a software decoder. The packets include 
timing, program flow information (e.g. branch targets, branch taken/not taken indications) and program-induced 
mode related information (e.g. Intel TSX state transitions, CR3 changes). These packets may be buffered internally 
before being sent to the memory subsystem or other output mechanism available in the platform. Debug software 
can process the trace data and reconstruct the program flow. 


Intel Processor Trace was first introduced in Intel® processors based on Broadwell microarchitecture and Intel 
Atom® processors based on Goldmont microarchitecture. Later generations include additional trace sources, 
including software trace instrumentation using PTWRITE, and Power Event tracing. 


31.1.1. Features and Capabilities 


Intel PT’s control flow trace generates a variety of packets that, when combined with the binaries of a program by 
a post-processing tool, can be used to produce an exact execution trace. The packets record flow information such 
as instruction pointers (IP), indirect branch targets, and directions of conditional branches within contiguous code 
regions (basic blocks). 


Intel PT can also be configured to log software-generated packets using PTWRITE, and packets describing 
processor power management events. Further, Precise Event-Based Sampling (PEBS) can be configured to log 
PEBS records in the Intel PT trace; see Section 18.5.5.2. 


In addition, the packets record other contextual, timing, and bookkeeping information that enables both functional 
and performance debugging of applications. Intel PT has several control and filtering capabilities available to 
customize the tracing information collected and to append other processor state and timing information to enable 
debugging. For example, there are modes that allow packets to be filtered based on the current privilege level 
(CPL) or the value of CR3. 


Configuration of the packet generation and filtering capabilities are programmed via a set of MSRs. The MSRs 
generally follow the naming convention of IA32_RTIT_*. The capability provided by these configuration MSRs are 
enumerated by CPUID, see Section 31.3. Details of the MSRs for configuring Intel PT are described in Section 
31.2.7. 


31.1.1.1 Packet Summary 


After a tracing tool has enabled and configured the appropriate MSRs, the processor will collect and generate trace 
information in the following categories of packets (for more details on the packets, see Section 31.4): 
¢ Packets about basic information on program execution; these include: 

— Packet Stream Boundary (PSB) packets: PSB packets act as ‘heartbeats’ that are generated at regular 
intervals (e.g., every 4K trace packet bytes). These packets allow the packet decoder to find the packet 
boundaries within the output data stream; a PSB packet should be the first packet that a decoder looks for 
when beginning to decode a trace. 


— Paging Information Packet (PIP): PIPs record modifications made to the CR3 register. This information, 
along with information from the operating system on the CR3 value of each process, allows the debugger 
to attribute linear addresses to their correct application source. 


— Time-Stamp Counter (TSC) packets: TSC packets aid in tracking wall-clock time, and contain some portion 
of the software-visible time-stamp counter. 


— Core Bus Ratio (CBR) packets: CBR packets contain the core:bus clock ratio. 
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Mini Time Counter (MTC) packets: MTC packets provide periodic indication of the passing of wall-clock time. 


Cycle Count (CYC) packets: CYC packets provide indication of the number of processor core clock cycles 
that pass between packets. 


Overflow (OVF) packets: OVF packets are sent when the processor experiences an internal buffer overflow, 
resulting in packets being dropped. This packet notifies the decoder of the loss and can help the decoder to 
respond to this situation. 


¢ Packets about control flow information: 


Taken Not-Taken (TNT) packets: TNT packets track the “direction” of direct conditional branches (taken or 
not taken). 


Target IP (TIP) packets: TIP packets record the target IP of indirect branches, exceptions, interrupts, and 
other branches or events. These packets can contain the IP, although that IP value may be compressed by 
eliminating upper bytes that match the last IP. There are various types of TIP packets; they are covered in 
more detail in Section 31.4.2.2. 


Flow Update Packets (FUP): FUPs provide the source IP addresses for asynchronous events (interrupt and 
exceptions), as well as other cases where the source address cannot be determined from the binary. 


MODE packets: These packets provide the decoder with important processor execution information so that 
it can properly interpret the dis-assembled binary and trace log. MODE packets have a variety of formats 
that indicate details such as the execution mode (16-bit, 32-bit, or 64-bit). 


¢ Packets inserted by software: 


PTWRITE (PTW) packets: includes the value of the operand passed to the PTWRITE instruction (see 
“PTWRITE - Write Data to a Processor Trace Packet” in Intel® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 2B). 


¢ Packets about processor power management events: 


MWAIT packets: Indicate successful completion of an MWAIT operation to a C-state deeper than CO.0. 
Power State Entry (PWRE) packets: Indicate entry to a C-state deeper than CO.0. 
Power State Exit (PWRX) packets: Indicate exit from a C-state deeper than CO.0, returning to CO. 


Execution Stopped (EXSTOP) packets: Indicate that software execution has stopped, due to events such as 
P-state change, C-state change, or thermal throttling. 


¢ Packets containing groups of processor state values: 


Block Begin Packets (BBP): Indicate the type of state held in the following group. 
Block Item Packets (BIP): Indicate the state values held in the group. 
Block End Packets (BEP): Indicate the end of the current group. 


31.2 INTEL® PROCESSOR TRACE OPERATIONAL MODEL 

This section describes the overall Intel Processor Trace mechanism and the essential concepts relevant to how it 
operates. 

31.2.1 Change of Flow Instruction (COFI) Tracing 


A basic program block is a section of code where no jumps or branches occur. The instruction pointers (IPs) in this 
block of code need not be traced, as the processor will execute them from start to end without redirecting code 
flow. Instructions such as branches, and events such as exceptions or interrupts, can change the program flow. 
These instructions and events that change program flow are called Change of Flow Instructions (COFI). There are 
three categories of COFI: 


° Direct transfer COFI. 
¢ Indirect transfer COFI. 
° Far transfer COFI. 
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The following subsections describe the COFI events that result in trace packet generation. Table 31-1 lists branch 
instruction by COFI types. For detailed description of specific instructions, see Intel® 64 and IA-32 Architectures 
Software Developer’s Manual. 

Table 31-1. COFI Type for Branch Instructions 
COFI Type Instructions 


Conditional Branch JA, JAE, JB, IBE, JC, JCXZ, JECXZ, JRCXZ, JE, JG, JGE, JL, JLE, JNA, JNAE, JNB, JNBE, JNC, JNE, JNG, JNGE, JNL, 
JNLE, JNO, JNP, JNS, JNZ, JO, JP, JPE, JPO, JS, JZ, LOOP, LOOPE, LOOPNE, LOOPNZ, LOOPZ 


Unconditional Direct Branch | JMP (€9 xx, EB xx), CALL (E8 xx) 
Indirect Branch JMP (FF /4), CALL (FF /2), RET (C3, C2 xx) 


Far Transfers INT1, INT3, INT n, INTO, IRET, IRETD, IRETQ, JMP (EA xx, FF /5), CALL (9A xx, FF /3), RET (CB, CA xx), 
SYSCALL, SYSRET, SYSENTER, SYSEXIT, VMLAUNCH, VMRESUME 


31.2.1.1 Direct Transfer COFI 


Direct Transfer COFI are relative branches. This means that their target is an IP whose offset from the current IP is 
embedded in the instruction bytes. It is not necessary to indicate target of these instructions in the trace output 
since it can be obtained through the source disassembly. Conditional branches need to indicate only whether the 
branch is taken or not. Unconditional branches do not need any recording in the trace output. There are two sub- 
categories: 


¢ Conditional Branch (Jcc, J*CXZ) and LOOP 


To track this type of instruction, the processor encodes a single bit (taken or not taken — TNT) to indicate the 
program flow after the instruction. 


Jcc, J*CXZ, and LOOP can be traced with TNT bits. To improve the trace packet output efficiency, the processor 
will compact several TNT bits into a single packet. 


¢ Unconditional Direct Jumps 


There is no trace output required for direct unconditional jumps (like JMP near relative or CALL near relative) 
since they can be directly inferred from the application assembly. Direct unconditional jumps do not generate a 
TNT bit or a Target IP packet, though TIP.PGD and TIP.PGE packets can be generated by unconditional direct 
jumps that toggle Intel PT enables (see Section 31.2.5). 


31.2.1.2 Indirect Transfer COFI 


Indirect transfer instructions involve updating the IP from a register or memory location. Since the register or 
memory contents can vary at any time during execution, there is no way to know the target of the indirect transfer 
until the register or memory contents are read. As a result, the disassembled code is not sufficient to determine the 
target of this type of COFI. Therefore, tracing hardware must send out the destination IP in the trace packet for 
debug software to determine the target address of the COFI. Note that this IP may be a linear or effective address 
(see Section 31.3.1.1). 


An indirect transfer instruction generates a Target IP Packet (TIP) that contains the target address of the branch. 
There are two sub-categories: 


¢ Near JMP Indirect and Near Call Indirect 


As previously mentioned, the target of an indirect COFI resides in the contents of either a register or memory 
location. Therefore, the processor must generate a packet that includes this target address to allow the 
decoder to determine the program flow. 

° Near RET 
When a CALL instruction executes, it pushes onto the stack the address of the next instruction following the 
CALL. Upon completion of the call procedure, the RET instruction is often used to pop the return address off of 
the call stack and redirect code flow back to the instruction following the CALL. 


A RET instruction simply transfers program flow to the address it popped off the stack. Because a called 
procedure may change the return address on the stack before executing the RET instruction, debug software 
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can be misled if it assumes that code flow will return to the instruction following the last CALL. Therefore, 
even for near RET, a Target IP Packet may be sent. 


— RET Compression 


A special case is applied if the target of the RET is consistent with what would be expected from tracking the 
CALL stack. If it is assured that the decoder has seen the corresponding CALL (with “corresponding” defined 
as the CALL with matching stack depth), and the RET target is the instruction after that CALL, the RET 
target may be “compressed”. In this case, only a single TNT bit of “taken” is generated instead of a Target 
IP Packet. To ensure that the decoder will not be confused in cases of RET compression, only RETs that 
correspond to CALLs which have been seen since the last PSB packet may be compressed in a given logical 
processor. For details, see “Indirect Transfer Compression for Returns (RET)” in Section 31.4.2.2. 


31.2.1.3. far Transfer COFI 


All operations that change the instruction pointer and are not near jumps are “far transfers”. This includes excep- 
tions, interrupts, traps, TSX aborts, and instructions that do far transfers. 


All far transfers will produce a Target IP (TIP) packet, which provides the destination IP address. For those far 
transfers that cannot be inferred from the binary source (e.g., asynchronous events such as exceptions and inter- 
rupts), the TIP will be preceded by a Flow Update packet (FUP), which provides the source IP address at which the 
event was taken. Table 31-24 indicates exactly which IP will be included in the FUP generated by a far transfer. 


31.2.2 Software Trace Instrumentation with PTWRITE 


PTWRITE provides a mechanism by which software can instrument the Intel PT trace. PTWRITE is a ring3-acces- 
sible instruction that can be passed to a register or memory variable, see “PTWRITE - Write Data to a Processor 
Trace Packet” in Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2B for details. The 
contents of that variable will be used as the payload for the PTW packet (see Table 31-41 “PTW Packet Definition”), 
inserted at the time of PTWRITE retirement, assuming PTWRITE is enabled and all other filtering conditions are 
met. Decode and analysis software will then be able to determine the meaning of the PTWRITE packet based on the 
IP of the associated PTWRITE instruction. 


PTWRITE is enabled via IA32_RTIT_CTL.PTWEn[12] (see Table 31-6). Optionally, the user can use 
IA32_RTIT_CTL.FUPonPTW[5] to enable PTW packets to be followed by FUP packets containing the IP of the asso- 
ciated PTWRITE instruction. Support for PTWRITE is introduced in Intel® Atom™ processors based on the Goldmont 
Plus microarchitecture. 


31.2.3. Power Event Tracing 


Power Event Trace is a capability that exposes core- and thread-level sleep state and power down transition infor- 
mation. When this capability is enabled, the trace will expose information about: 


— Scenarios where software execution stops. 
¢ Due to sleep state entry, frequency change, or other powerdown. 
¢ Includes the IP, when in the tracing context. 
— The requested and resolved hardware thread C-state. 
¢ Including indication of hardware autonomous C-state entry. 
— The last and deepest core C-state achieved during a sleep session. 
— The reason for C-state wake. 


This information is in addition to the bus ratio (CBR) information provided by default after any powerdown, and the 
timing information (TSC, TMA, MTC, CYC) provided during or after a powerdown state. 


Power Event Trace is enabled via IA32_RTIT_CTL.PwrEvtEn[4]. Support for Power Event Tracing is introduced in 
Intel® Atom™ processors based on the Goldmont Plus microarchitecture. 
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31.2.4 ‘Trace Filtering 


Intel Processor Trace provides filtering capabilities, by which the debug/profile tool can control what code is traced. 


31.2.4.1 Filtering by Current Privilege Level (CPL) 


Intel PT provides the ability to configure a logical processor to generate trace packets only when CPL = 0, when 
CPL > 0, or regardless of CPL. 


CPL filtering ensures that no IPs or other architectural state information associated with the filtered CPL can be 
seen in the log. For example, if the processor is configured to trace only when CPL > 0, and software executes 
SYSCALL (changing the CPL to 0), the destination IP of the SYSCALL will be suppressed from the generated packet 
(see the discussion of TIP.PGD in Section 31.4.2.5). 


It should be noted that CPL is always 0 in real-address mode and that CPL is always 3 in virtual-8086 mode. To 
trace code in these modes, filtering should be configured accordingly. 


When software is executing in a non-enabled CPL, ContextEn is cleared. See Section 31.2.5.1 for details. 


31.2.4.2 Filtering by CR3 


Intel PT supports a CR3-filtering mechanism by which the generation of packets containing architectural states can 
be enabled or disabled based on the value of CR3. A debugger can use CR3 filtering to trace only a single applica- 
tion without context switching the state of the RTIT MSRs. For the reconstruction of traces from software with 
multiple threads, debug software may wish to context-switch for the state of the RTIT MSRs (if the operating 
system does not provide context-switch support) to separate the output for the different threads (see Section 
31.3.5, “Context Switch Consideration”). 


To trace for only a single CR3 value, software can write that value to the IA32_RTIT_CR3_MATCH MSR, and set 
IA32_RTIT_CTL.CR3Filter. When CR3 value does not match IA32_RTIT_CR3_MATCH and IA32_RTIT_CTL.CR3Filter 
is 1, ContextEn is forced to 0, and packets containing architectural states will not be generated. Some other 
packets can be generated when ContextEn is 0; see Section 31.2.5.3 for details. When CR3 does match 
IA32_RTIT_CR3_MATCH (or when IA32_RTIT_CTL.CR3Filter is 0), CR3 filtering does not force ContextEn to 0 
(although it could be 0 due to other filters or modes). 


CR3 matches IA32_RTIT_CR3_MATCH if the two registers are identical for bits 63:12, or 63:5 when in PAE paging 
mode; the lower 5 bits of CR3 and IA32_RTIT_CR3_MATCH are ignored. CR3 filtering is independent of the value 
of CRO.PG. 


When CR3 filtering is in use, PIP packets may still be seen in the log if the processor is configured to trace when 
CPL = 0 (IA32_RTIT_CTL.OS = 1). If not, no PIP packets will be seen. 


31.2.4.3 Filtering by IP 


Trace packet generation with configurable filtering by IP is supported if CPUID.(EAX=14H, ECX=0):EBX[bit 2] = 1. 
Intel PT can be configured to enable the generation of packets containing architectural states only when the 
processor is executing code within certain IP ranges. If the IP is outside of these ranges, generation of some 
packets is blocked. 


IP filtering is enabled using the ADDRn_CFG fields in the IA32_RTIT_CTL MSR (Section 31.2.7.2), where the digit 
'n' is a zero-based number that selects which address range is being configured. Each ADDRn_CFG field configures 
the use of the register pair IA32_RTIT_ADDRn_A and IA32_RTIT_ADDRn_B (Section 31.2.7.5). 
IA32_RTIT_ADDRn_A defines the base and IA32_RTIT_ADDRn_B specifies the limit of the range in which tracing is 
enabled. Thus each range, referred to as the ADDRn range, is defined by [I[A32_RTIT_ADDRn_A, 
IA32_RTIT_ADDRn_B]. There can be multiple such ranges, software can query CPUID (Section 31.3.1) for the 
number of ranges supported on a processor. 


Default behavior (ADDRn_CFG=0) defines no IP filter range, meaning FilterEn is always set. In this case code at 
any IP can be traced, though other filters, such as CR3 or CPL, could limit tracing. When ADDRn_CFG is set to 
enable IP filtering (see Section 31.3.1), tracing will commence when a taken branch or event is seen whose target 
address is in the ADDRn range. 


While inside a tracing region and with FilterEn is set, leaving the tracing region may only be detected once a taken 
branch or event with a target outside the range is retired. If an ADDRn range is entered or exited by executing the 
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next sequential instruction, rather than by a control flow transfer, FilterEn may not toggle immediately. See Section 
31.2.5.5 for more details on FilterEn. 


Note that these address range base and limit values are inclusive, such that the range includes the first and last 
instruction whose first instruction byte is in the ADDRn range. 


Depending upon processor implementation, IP filtering may be based on linear or effective address. This can cause 
different behavior between implementations if CSbase is not equal to zero or in real mode. See Section 31.3.1.1 for 
details. Software can query CPUID to determine filters are based on linear or effective address (Section 31.3.1). 


Note that some packets, such as MTC (Section 31.3.7) and other timing packets, do not depend on FilterEn. For 
details on which packets depend on FilterEn, and hence are impacted by IP filtering, see Section 31.4.1. 


TraceStop 


The ADDRn ranges can also be configured to cause tracing to be disabled upon entry to the specified region. This is 
intended for cases where unexpected code is executed, and the user wishes to immediately stop generating 
packets in order to avoid overwriting previously written packets. 


The TraceStop mechanism works much the same way that IP filtering does, and uses the same address comparison 
logic. The TraceStop region base and limit values are programmed into one or more ADDRn ranges, but 
IA32_RTIT_CTL.ADDRn_CFG is configured with the TraceStop encoding. Like FilterEn, TraceStop is detected when 
a taken branch or event lands in a TraceStop region. 

Further, TraceStop requires that TriggerEn=1 at the beginning of the branch/event, and ContextEn=1 upon 
completion of the branch/event. When this happens, the CPU will set IA32_RTIT_STATUS.Stopped, thereby 
clearing TriggerEn and hence disabling packet generation. This may generate a TIP.PGD packet with the target IP 
of the branch or event that entered the TraceStop region. Finally, a TraceStop packet will be inserted, to indicate 
that the condition was hit. 

If a TraceStop condition is encountered during buffer overflow (Section 31.3.8), it will not be dropped, but will 
instead be signaled once the overflow has resolved. 

Note that a TraceStop event does not guarantee that all internally buffered packets are flushed out of internal 
buffers. To ensure that this has occurred, the user should clear TraceEn. 

To resume tracing after a TraceStop event, the user must first disable Intel PT by clearing IA32_RTIT_CTL.TraceEn 
before the IA32_RTIT_STATUS.Stopped bit can be cleared. At this point Intel PT can be reconfigured, and tracing 
resumed. 


Note that the IA32_RTIT_STATUS.Stopped bit can also be set using the ToPA STOP bit. See Section 31.2.6.2. 


IP Filtering Example 


The following table gives an example of IP filtering behavior. Assume that IA32_RTIT_ADDRn_A = the IP of Range- 
Base, and that IA32_RTIT_ADDRn_B = the IP of RangeLimit, while IA32_RTIT_CTL.ADDRn_CFG = Oxi (enable 
ADDRnh range as a FilterEn range). 


Table 31-2. IP Filtering Packet Example 


Code Flow Packets 
Bar: TIP.PGE(RangeBase) 
jmp RangeBase // jump into filter range TNT(0) 
RAG aEASe. TIP.PGD(RangeLimit+1) 


jcc Foo // not taken 
add eax, 1 

Foo: 
jmp RangeLimit+1 // jump out of filter range 
RangeLimit: 


jec Bar 
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IP Filtering and TraceStop 


It is possible for the user to configure IP filter range(s) and TraceStop range(s) that overlap. In this case, code 
executing in the non-overlapping portion of either range will behave as would be expected from that range. Code 
executing in the overlapping range will get TraceStop behavior. 


31.2.5 Packet Generation Enable Controls 


Intel Processor Trace includes a variety of controls that determine whether a packet is generated. In general, most 
packets are sent only if Packet Enable (PacketEn) is set. PacketEn is an internal state maintained in hardware in 
response to software configurable enable controls, PacketEn is not visible to software directly. The relationship of 
PacketEn to the software-visible controls in the configuration MSRs is described in this section. 


31.2.5.1 Packet Enable (PacketEn) 


When PacketEn is set, the processor is in the mode that Intel PT is monitoring. PacketEn is composed of other 
states according to this relationship: 


PacketEn := TriggerEn AND ContextEn AND FilterEn AND BranchEn 


These constituent controls are detailed in the following subsections. 


PacketEn ultimately determines when the processor is tracing. When PacketEn is set, all control flow packets are 
enabled. When PacketEn is clear, no control flow packets are generated, though other packets (timing and book- 
keeping packets) may still be sent. See Section 31.2.6 for details of PacketEn and packet generation. 


Note that, on processors that do not support IP filtering (i.e., CPUID.(EAX=14H, ECX=0):EBX[bit 2] = 0), FilterEn 
is treated as always Set. 


31.2.5.2 Trigger Enable (TriggerEn) 


Trigger Enable (TriggerEn) is the primary indicator that trace packet generation is active. TriggerEn is set when 
IA32_RTIT_CTL.TraceEn is set, and cleared by any of the following conditions: 

¢® TraceEn is cleared by software. 

¢ A TraceStop condition is encountered and IA32_RTIT_STATUS.Stopped is set. 

¢ JA32_RTIT_STATUS.Error is set due to an operational error (see Section 31.3.9). 


Software can discover the current TriggerEn value by reading the IA32_RTIT_STATUS. TriggerEn bit. When Trig- 
gerEn is clear, tracing is inactive and no packets are generated. 


31.2.5.3. Context Enable (ContextEn) 


Context Enable (ContextEn) indicates whether the processor is in the state or mode that software configured 
hardware to trace. For example, if execution with CPL = 0 code is not being traced (IA32_RTIT_CTL.OS = 0), then 
ContextEn will be 0 when the processor is in CPLO. 


Software can discover the current ContextEn value by reading the IA32_RTIT_STATUS.ContextEn bit. ContextEn is 
defined as follows: 


ContextEn = !((IA32_RTIT_CTL.OS = 0 AND CPL = 0) OR 
(IA32_ RTIT_CTL.USER = 0 AND CPL > 0) OR (IS_IN_A PRODUCTION ENCLAVE’) OR 
(IA32_RTIT_CTL.CR3Filter = 1 AND IA32 RTIT_CR3_ MATCH does not match CR3) 


If the clearing of ContextEn causes PacketEn to be cleared, a Packet Generation Disable (TIP.PGD) packet is gener- 
ated, but its IP payload is suppressed. If the setting of ContextEn causes PacketEn to be set, a Packet Generation 
Enable (TIP.PGE) packet is generated. 


When ContextEn is 0, control flow packets (TNT, FUP, TIP.*, MODE.*) are not generated, and no Linear Instruction 
Pointers (LIPs) are exposed. However, some packets, such as MTC and PSB (see Section 31.4.2.16 and Section 


1. Trace packets generation is disabled in a production enclave, see Section 31.2.8.5. See Intel® Software Guard 
Extensions Programming Reference about differences between a production enclave and a debug enclave. 
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31.4.2.17), may still be generated while ContextEn is 0. For details of which packets are generated only when 
ContextEn is set, see Section 31.4.1. 


The processor does not update ContextEn when TriggerEn = 0. 
The value of ContextEn will toggle only when TriggerEn = 1. 


31.2.5.4 Branch Enable (BranchEn) 


This value is based purely on the JA32_RTIT_CTL.BranchEn value. If BranchEn is not set, then relevant COFI 
packets (TNT, TIP*, FUP, MODE.*) are suppressed. Other packets related to timing (TSC, TMA, MTC, CYC), as well 
as PSB, will be generated normally regardless. Further, PIP and VMCS continue to be generated, as indicators of 
what software is running. 


31.2.5.5 Filter Enable (FilterEn) 


Filter Enable indicates that the Instruction Pointer (IP) is within the range of IPs that Intel PT is configured to watch. 
Software can get the state of Filter Enable by a RDMSR of IA32_RTIT_STATUS.FilterEn. For details on configuration 
and use of IP filtering, see Section 31.2.4.3. 


On clearing of FilterEn that also clears PacketEn, a Packet Generation Disable (TIP.PGD) will be generated, but 
unlike the ContextEn case, the IP payload may not be suppressed. For direct, unconditional branches, as well as for 
indirect branches (including RETs), the PGD generated by leaving the tracing region and clearing FilterEn will 
contain the target IP. This means that IPs from outside the configured range can be exposed in the trace, as long 
as they are within context. 


When FilterEn is 0, control flow packets are not generated (e.g., TNT, TIP). However, some packets, such as PIP, 
MTC, and PSB, may still be generated while FilterEn is clear. For details on packet enable dependencies, see Section 
31.4.1. 


After TraceEn is set, FilterEn is set to 1 at all times if there is no IP filter range configured by software 
(IA32_RTIT_CTL.ADDRn_CFG != 1, for all n), or if the processor does not support IP filtering (i-.e., 
CPUID.(EAX=14H, ECX=0):EBX[bit 2] = 0). FilterEn will toggle only when TraceEn=1 and ContextEn=1, and when 
at least one range is configured for IP filtering. 


31.2.6 Trace Output 


Intel PT output should be viewed independently from trace content and filtering mechanisms. The options available 
for trace output can vary across processor generations and platforms. 


Trace output is written out using one of the following output schemes, as configured by the ToPA and FabricEn bit 
fields of IA32_RTIT_CTL (see Section 31.2.7.2): 


¢ Asingle, contiguous region of physical address space. 


¢ Acollection of variable-sized regions of physical memory. These regions are linked together by tables of 
pointers to those regions, referred to as Table of Physical Addresses (TOPA). The trace output stores bypass 
the caches and the TLBs, but are not serializing. This is intended to minimize the performance impact of the 
output. 


° A platform-specific trace transport subsystem. 


Regardless of the output scheme chosen, Intel PT stores bypass the processor caches by default. This ensures that 
they don't consume precious cache space, but they do not have the serializing aspects associated with un-cache- 
able (UC) stores. Software should avoid using MTRRs to mark any portion of the Intel PT output region as UC, as 
this may override the behavior described above and force Intel PT stores to UC, thereby incurring severe perfor- 
mance impact. 


There is no guarantee that a packet will be written to memory or other trace endpoint after some fixed number of 
cycles after a packet-producing instruction executes. The only way to assure that all packets generated have 
reached their endpoint is to clear TraceEn and follow that with a store, fence, or serializing instruction; doing so 
ensures that all buffered packets are flushed out of the processor. 
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31.2.6.1 Single Range Output 


When IA32_RTIT_CTL.ToPA and IA32_RTIT_CTL.FabricEn bits are clear, trace packet output is sent to a single, 
contiguous memory (or MMIO if DRAM is not available) range defined by a base address in 
IA32_RTIT_OUTPUT_BASE (Section 31.2.7.7) and mask value in IA32_RTIT_OUTPUT_MASK_PTRS (Section 
31.2.7.8). The current write pointer in this range is also stored in IA32_RTIT_OUTPUT_MASK_PTRS. This output 
range is circular, meaning that when the writes wrap around the end of the buffer they begin again at the base 
address. 


This output method is best suited for cases where Intel PT output is either: 
* Configured to be directed to a sufficiently large contiguous region of DRAM. 


* Configured to go to an MMIO debug port, in order to route Intel PT output to a platform-specific trace endpoint 
(e.g., JTAG). In this scenario, a specific range of addresses is written in a circular manner, and SoC will intercept 
these writes and direct them to the proper device. Repeated writes to the same address do not overwrite each 
other, but are accumulated by the debugger, and hence no data is lost by the circular nature of the buffer. 


The processor will determine the address to which to write the next trace packet output byte as follows: 


OutputBase [63:0] 


IA32_RTIT OUTPUT BASE[63:0] 


OutputMask[63:0] := ZeroExtend6é4 (IA32 RTIT OUTPUT MASK PTRS [31:0] ) 
OutputOffset [63:0] := ZeroExtend64 (IA32 RTIT OUTPUT MASK PTRS [63:32] ) 
trace store phys addr := (OutputBase & ~OutputMask) + (OutputOffset & OutputMask) 


Single-Range Output Errors 
If the output base and mask are not properly configured by software, an operational error (see Section 31.3.9) will 
be signaled, and tracing disabled. Error scenarios with single-range output are: 
¢ ~=Mask value is non-contiguous. 
IA32_RTIT_OUTPUT_MASK_PTRS.MaskOrTablePointer value has a 0 in a less significant bit position than the 
most significant bit containing a 1. 
¢ Base address and Mask are mis-aligned, and have overlapping bits set. 
IA32_RTIT_OUTPUT_BASE && IA32_RTIT_OUTPUT_MASK_PTRS[31:0] > 0. 
° Illegal Output Offset 


IA32_RTIT_OUTPUT_MASK_PTRS.OutputOffset is greater than the mask value 
IA32_RTIT_OUTPUT_MASK_PTRS[31:0]. 


Also note that errors can be signaled due to trace packet output overlapping with restricted memory, see Section 
31.2.6.4. 


31.2.6.2 Table of Physical Addresses (ToPA) 


When JA32_RTIT_CTL.ToPA is set and IA32_RTIT_CTL.FabricEn is clear, the ToPA output mechanism is utilized. The 
ToPA mechanism uses a linked list of tables; see Figure 31-1 for an illustrative example. Each entry in the table 
contains some attribute bits, a pointer to an output region, and the size of the region. The last entry in the table 
may hold a pointer to the next table. This pointer can either point to the top of the current table (for circular array) 
or to the base of another table. The table size is not fixed, since the link to the next table can exist at any entry. 


The processor treats the various output regions referenced by the ToPA table(s) as a unified buffer. This means that 
a single packet may span the boundary between one output region and the next. 


The ToPA mechanism is controlled by three values maintained by the processor: 


° proc_trace_table_base. 
This is the physical address of the base of the current ToPA table. When tracing is enabled, the processor loads 
this value from the IA32_RTIT_OUTPUT_BASE MSR. While tracing is enabled, the processor updates the 
IA32_RTIT_OUTPUT_BASE MSR with changes to proc_trace_table_base, but these updates may not be 
synchronous to software execution. When tracing is disabled, the processor ensures that the MSR contains the 
latest value of proc_trace_table_base. 
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°* proc_trace_table_offset. 
This indicates the entry of the current table that is currently in use. (This entry contains the address of the 
current output region.) When tracing is enabled, the processor loads the value from bits 31:7 (MaskOrT- 
ableOffset) of the IA32_RTIT_OUTPUT_MASK_PTRS into bits 27:3 of proc_trace_table_offset. While tracing is 
enabled, the processor updates IA32_RTIT_OUTPUT_MASK_PTRS.MaskOrTableOffset with changes to 
proc_trace_table_offset, but these updates may not be synchronous to software execution. When tracing is 
disabled, the processor ensures that the MSR contains the latest value of proc_trace_table_offset. 


°* proc_trace_output_offset. 
This a pointer into the current output region and indicates the location of the next write. When tracing is 
enabled, the processor loads this value from bits 63:32 (OutputOffset) of the 
IA32_RTIT_OUTPUT_MASK_PTRS. While tracing is enabled, the processor updates 
IA32_RTIT_OUTPUT_MASK_PTRS.OutputOffset with changes to proc_trace_output_offset, but these updates 
may not be synchronous to software execution. When tracing is disabled, the processor ensures that the MSR 
contains the latest value of proc_trace_output_offset. 


Figure 31-1 provides an illustration (not to scale) of the table and associated pointers. 


OFF_FFFF _FFFFH 
Physical Memory 


proc_trace_output_offset: IA32_RTIT_OUTPUT_MASK_PTRS.OutputOffset 


TableBaseB 
OutputBaseY 


OutputBaseX |~<@t——— proc_trace_table_offset: 
IA32_RTIT_OUTPUT_MASK_PRS.MaskOrTableOffset<<3 


~<a proc_trace_table_base: |A32_RTIT_OUTPUT_BASE 


Figure 31-1. ToPA Memory Illustration 


With the ToPA mechanism, the processor writes packets to the current output region (identified by 
proc_trace_table_base and the proc_trace_table_offset). The offset within that region to which the next byte will 
be written is identified by proc_trace_output_offset. When that region is filled with packet output (thus 
proc_trace_output_offset = RegionSize-1), proc_trace_table_offset is moved to the next ToPA entry, 
proc_trace_output_offset is set to 0, and packet writes begin filling the new output region specified by 
proc_trace_table_offset. 


As packets are written out, each store derives its physical address as follows: 


trace store phys addr := Base address from current ToPA table entry + 
proc_trace output _offset 


Eventually, the regions represented by all entries in the table may become full, and the final entry of the table is 
reached. An entry can be identified as the final entry because it has either the END or STOP attribute. The END 
attribute indicates that the address in the entry does not point to another output region, but rather to another ToPA 
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table. The STOP attribute indicates that tracing will be disabled once the corresponding region is filled. See Table 
31-3 and the section that follows for details on STOP. 


When an END entry is reached, the processor loads proc_trace_table_base with the base address held in this END 
entry, thereby moving the current table pointer to this new table. The proc_trace_table_offset is reset to 0, as is 
the proc_trace_output_offset, and packet writes will resume at the base address indicated in the first entry. 


If the table has no STOP or END entry, and trace-packet generation remains enabled, eventually the maximum 
table size will be reached (proc_trace_table_offset = OFFFFFF8H). In this case, the proc_trace_table_offset and 
proc_trace_output_offset are reset to 0 (wrapping back to the beginning of the current table) once the last output 
region is filled. 


It is important to note that processor updates to the IA32_RTIT_OUTPUT_BASE and 
IA32_RTIT_OUTPUT_MASK_PTRS MSRs are asynchronous to instruction execution. Thus, reads of these MSRs 
while Intel PT is enabled may return stale values. Like all IA32_RTIT_* MSRs, the values of these MSRs should not 
be trusted or saved unless trace packet generation is first disabled by clearing IA32_RTIT_CTL.TraceEn. This 
ensures that the output MSR values account for all packets generated to that point, after which the processor will 
cease updating the output MSR values until tracing resumes. 


The processor may cache internally any number of entries from the current table or from tables that it references 
(directly or indirectly). If tracing is enabled, the processor may ignore or delay detection of modifications to these 
tables. To ensure that table changes are detected by the processor in a predictable manner, software should clear 
TraceEn before modifying the current table (or tables that it references) and only then re-enable packet genera- 
tion. 


Single Output Region ToPA Implementation 


The first processor generation to implement Intel PT supports only ToPA configurations with a single ToPA entry 
followed by an END entry that points back to the first entry (creating one circular output buffer). Such processors 
enumerate CPUID.(EAX=14H,ECX=0):ECX.MENTRY[bit 1] = 0 and CPUID.(EAX=14H,ECX=0):ECX.TOPAOUT[bit 
0) =1. 

If CPUID.(EAX=14H,ECX=0):ECX.MENTRY[bit 1] = 0, ToPA tables can hold only one output entry, which must be 
followed by an END=1 entry which points back to the base of the table. Hence only one contiguous block can be 
used as output. 


The lone output entry can have INT or STOP set, but nonetheless must be followed by an END entry as described 
above. Note that, if INT=1, the PMI will actually be delivered before the region is filled. 


ToPA Table Entry Format 


The format of ToPA table entries is shown in Figure 31-2. The size of the address field is determined by the 
processor’s physical-address width (MAXPHYADDR) in bits, as reported in CPUID.80000008H:EAX[7:0]. 


63 MAXPHYADDR-1 1211 109 65 43 210 


Output Region Base Physical Address 


9:6 Size 
4: STOP 


2: INT Reserved 
0:END 


Figure 31-2. Layout of ToPA Table Entry 


Table 31-3 describes the details of the ToPA table entry fields. If reserved bits are set to 1, an error is signaled. 


1. Although WRMSR is a serializing instruction, the execution of WRMSR that forces packet writes by clearing TraceEn does not itself 
cause these writes to be globally observed. 
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Table 31-3. ToPA Table Entry Fields 


ToPA Entry Field Description 

Output Region If END=O, this is the base physical address of the output region specified by this entry. Note that all regions 
Base Physical must be aligned based on their size. Thus a 2M region must have bits 20:12 clear. If the region is not properly 
Address aligned, an operational error will be signaled when the entry is reached. 


If END=1, this is the 4K-aligned base physical address of the next ToPA table (which may be the base of the cur- 
rent table, or the first table in the linked list if a circular buffer is desired). If the processor supports only a single 
ToPA output region (see above), this address must be the value currently in the [A32_RTIT_OUTPUT_BASE 
MSR. 


Size Indicates the size of the associated output region. Encodings are: 

0: 4K, 1: 8K, 2: 16K, 3: 32K, 4: 64K, 5:128K, 6:256K, 7:512K, 
8: 1M, 9: 2M, 10: 4M, 11: 8M, 12:16M, 13:32M, 14:64M,  15:128M 
This field is ignored if END=1. 


STOP When the output region indicated by this entry is filled, software should disable packet generation. This will be 
accomplished by setting [A32_RTIT_STATUS.Stopped, which clears TriggerEn. This bit must be 0 if END=1; oth- 
erwise it is treated as reserved bit violation (see ToPA Errors). 


INT When the output region indicated by this entry is filled, signal Perfmon LVT interrupt. 

Note that if both INT and STOP are set in the same entry, the STOP will happen before the INT. Thus the inter- 
rupt handler should expect that the [A32_RTIT_STATUS.Stopped bit will be set, and will need to be reset before 
tracing can be resumed. 

This bit must be 0 if END=1; otherwise it is treated as reserved bit violation (see ToPA Errors). 


END If set, indicates that this is an END entry, and thus the address field points to a table base rather than an output 
region base. 

If END=1, INT and STOP must be set to O; otherwise it is treated as reserved bit violation (see ToPA Errors). The 
Size field is ignored in this case. 

If the processor supports only a single ToPA output region (see above), END must be set in the second table 
entry. 


ToPA STOP 


Each ToPA entry has a STOP bit. If this bit is set, the processor will set the IA32_RTIT_STATUS.Stopped bit when 
the corresponding trace output region is filled. This will clear TriggerEn and thereby cease packet generation. See 
Section 31.2.7.4 for details on IA32_RTIT_STATUS.Stopped. This sequence is known as “ToPA Stop”. 


No TIP.PGD packet will be seen in the output when the ToPA stop occurs, since the disable happens only when the 
region is already full. When this occurs, output ceases after the last byte of the region is filled, which may mean 
that a packet is cut off in the middle. Any packets remaining in internal buffers are lost and cannot be recovered. 


When ToPA stop occurs, the IA32_RTIT_OUTPUT_BASE MSR will hold the base address of the table whose entry 
had STOP=1. IA32_RTIT_OUTPUT_MASK_PTRS.MaskOrTableOffset will hold the index value for that entry, and the 
IA32_RTIT_OUTPUT_MASK_PTRS.OutputOffset should be set to the size of the region minus one. 


Note that this means the offset pointer is pointing to the next byte after the end of the region, a configuration that 
would produce an operational error if the configuration remained when tracing is re-enabled with 
IA32_RTIT_STATUS.Stopped cleared. 


ToPA PMI 


Each ToPA entry has an INT bit. If this bit is set, the processor will signal a performance-monitoring interrupt (PMI) 
when the corresponding trace output region is filled. This interrupt is not precise, and it is thus likely that writes to 
the next region will occur by the time the interrupt is taken. 


The following steps should be taken to configure this interrupt: 


1. Enable PMI via the LVT Performance Monitor register (at MMIO offset 340H in xAPIC mode; via MSR 834H in 
x2APIC mode). See Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3B for more 
details on this register. For ToPA PMI, set all fields to 0, save for the interrupt vector, which can be selected by 
software. 


2. Set up an interrupt handler to service the interrupt vector that a TOPA PMI can raise. 
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3. Set the interrupt flag by executing STI. 


4. Set the INT bit in the ToPA entry of interest and enable packet generation, using the ToPA output option. Thus, 
TraceEn=ToPA=1 in the IA32_RTIT_CTL MSR. 


Once the INT region has been filled with packet output data, the interrupt will be signaled. This PMI can be distin- 
guished from others by checking bit 55 (Trace_ToPA_PMI) of the IA32_PERF_GLOBAL_STATUS MSR (MSR 38Eh). 
Once the ToPA PMI handler has serviced the relevant buffer, writing 1 to bit 55 of the MSR at 390H 
(IA32_GLOBAL_STATUS_RESET) clears IA32_PERF_GLOBAL_STATUS.Trace_ToPA_PMI. 


Intel PT is not frozen on PMI, and thus the interrupt handler will be traced (though filtering can prevent this). The 
Freeze_Perfmon_on_PMI and Freeze_LBRs_on_PMI settings in IA32_DEBUGCTL will be applied on ToPA PMI just as 
on other PMIs, and hence Perfmon counters are frozen. 


Assuming the PMI handler wishes to read any buffered packets for persistent output, or wishes to modify any Intel 
PT MSRs, software should first disable packet generation by clearing TraceEn. This ensures that all buffered packets 
are written to memory and avoids tracing of the PMI handler. The configuration MSRs can then be used to deter- 
mine where tracing has stopped. If packet generation is disabled by the handler, it should then be manually re- 
enabled before the IRET if continued tracing is desired. 


In rare cases, it may be possible to trigger a second ToPA PMI before the first is handled. This can happen if another 
ToPA region with INT=1 is filled before, or shortly after, the first PMI is taken, perhaps due to EFLAGS.IF being 
cleared for an extended period of time. This can manifest in two ways: either the second PMI is triggered before the 
first is taken, and hence only one PMI is taken, or the second is triggered after the first is taken, and thus will be 
taken when the handler for the first completes. Software can minimize the likelihood of the second case by clearing 
TraceEn at the beginning of the PMI handler. Further, it can detect such cases by then checking the Interrupt 
Request Register (IRR) for PMI pending, and checking the ToPA table base and off-set pointers (in 
IA32_RTIT_OUTPUT_BASE and IA32_RTIT_OUTPUT_MASK_PTRS) to see if multiple entries with INT=1 have been 
filled. 


PMI Preservation 


In some cases a TOPA PMI may be taken after completion of an XSAVES instruction that saves Intel PT state, and 
in such cases any modification of Intel PT MSRs within the PMI handler will not persist when the saved Intel PT 
context is later restored with XRSTORS. To account for such a scenario, the PMI Preservation feature has been 
added. Support for this feature is indicated by CPUID.(EAX=14H, ECX=0):EBX[bit 6]. 


When IA32_RTIT_CTL.InjectPsbPmiOnEnable[56] = 1, PMI preservation is enabled. When a ToPA region with 
INT=1 is filled, a PMI is pended and the new IA32_RTIT_STATUS.PendToPAPMI[7] is set to 1. If this bit is set when 
Intel PT is enabled, such that IA32_RTIT_CTL.TraceEn[0] transitions from 0 to 1, a ToPA PMI is pended. This 
behavior ensures that any ToPA PMI that is pended during XSAVES, and hence can't be properly handled, will be re- 
pended when the saved PT state is restored. 


When this feature is enabled, the PMI handler should take the following actions: 


1. Ignore ToPA PMIs that are taken when TraceEn = 0. This indicates that the PMI was pended during Intel PT 
disable, and the PendToPAPMI flag will ensure that the PMI is re-pended once Intel PT is re-enabled in the same 
context. For this reason, the PendToPAPMI bit should be left set to 1. 


2. If TraceEn=1 and the PMI can be properly handled, clear the new PendTopaPMI bit. This will ensure that 
additional, spurious ToPA PMIs are not taken. It is required that PendToPAPMI is cleared before the PMI LVT 
mask is cleared in the APIC, and before any clearing of either LBRS_ FROZEN or COUNTERS_FROZEN in 
IA32_PERF_GLOBAL_STATUS. 


ToPA PMI and Single Output Region ToPA Implementation 


A processor that supports only a single ToPA output region implementation (such that only one output region is 
supported; see above) will attempt to signal a TOPA PMI interrupt before the output wraps and overwrites the top 
of the buffer. To support this functionality, the PMI handler should disable packet generation as soon as possible. 


Due to PMI skid, it is possible that, in rare cases, the wrap will have occurred before the PMI is delivered. Software 
can avoid this by setting the STOP bit in the ToPA entry (see Table 31-3); this will disable tracing once the region is 
filled, and no wrap will occur. This approach has the downside of disabling packet generation so that some of the 
instructions that led up to the PMI will not be traced. If the PMI skid is significant enough to cause the region to fill 
and tracing to be disabled, the PMI handler will need to clear the IA32_RTIT_STATUS.Stopped indication before 
tracing can resume. 
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ToPA PMI and XSAVES/XRSTORS State Handling 


In some cases the ToPA PMI may be taken after completion of an XSAVES instruction that switches Intel PT state, 
and in such cases any modification of Intel PT MSRs within the PMI handler will not persist when the saved Intel PT 
context is later restored with XRSTORS. To account for such a scenario, it is recommended that the Intel PT output 
configuration be modified by altering the ToPA tables themselves, rather than the Intel PT output MSRs. On proces- 
sors that support PMI preservation (CPUID.(EAX=14H, ECX=0):EBX[bit 6] = 1), setting IA32_RTIT_CTL.InjectPsb- 
PmiOnEnable[56] = 1 will ensure that a PMI that is pending at the time PT is disabled will be recorded by setting 

IA32_RTIT_STATUS.PendTopaPMI[7] = 1. A PMI will then be pended when the saved PT context is later restored. 


Table 31-4 depicts a recommended PMI handler algorithm for managing multi-region ToPA output and handling 
ToPA PMIs that may arrive between XSAVES and XRSTORS, if PMI preservation is not in use. This algorithm is flex- 
ible to allow software to choose between adding entries to the current ToPA table, adding a new ToPA table, or using 
the current ToPA table as a circular buffer. It assumes that the ToPA entry that triggers the PMI is not the last entry 
in the table, which is the recommended treatment. 


Table 31-4. Algorithm to Manage Intel PT ToPA PMI and XSAVES/XRSTORS 


Pseudo Code Flow 


IF (IA32_ PERF GLOBAL STATUS .ToPA) 

Save IA32_RTIT CTL value; 
IF ( IA32_RTIT_CTL.TraceEN ) 
Disable Intel PT by clearing TraceEn; 
FI; 
IF ( there is space available to grow the current ToPA table ) 
Add one or more ToPA entries after the last entry in the ToPA table; 
Point new ToPA entry address field(s) to new output region base(s) ; 


ELSE 
Modify an upcoming ToPA entry in the current table to have END=1; 
IF (output should transition to a new ToPA table ) 
Point the address of the “END=1" entry of the current table to the new table base; 
ELSE 
/* Continue to use the current ToPA table, make a circular. */ 
Point the address of the “END=1"1 entry to the base of the current table; 
Modify the ToPA entry address fields for filled output regions to point to new, unused output regions; 
/* Filled regions are those with index in the range of 0 to (IA32_RTIT MASK PTRS.MaskOrTableOffset -1). */ 


FI; 
FI; 
Restore saved IA32_RTIT CTL.value; 
FI; 
ToPA Errors 


When a malformed ToPA entry is found, an operational error results (See Section 31.3.9). A malformed entry can 
be any of the following: 


1. ToPA entry reserved bit violation. 
This describes cases where a bit marked as reserved in Section 31.2.6.2 above is set to 1. 


2. ToPA alignment violation. 
This includes cases where illegal ToPA entry base address bits are set to 1: 


a. ToPA table base address is not 4KB-aligned. The table base can be from a WRMSR to 
IA32_RTIT_OUTPUT_BASE, or from a ToPA entry with END=1. 


b. ToPA entry base address is not aligned to the ToPA entry size (e.g., a 2MB region with base address[20:12] 
not equal to 0), for ToPA entries with END=0O. 


c. TOPA entry base address sets upper physical address bits not supported by the processor. 
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3. Illegal ToPA Output Offset. 
IA32_RTIT_OUTPUT_MASK_PTRS.OutputOffset is greater than or equal to the size of the current ToPA output 
region size. 


4. ToPA rules violations. 
These are similar to ToPA entry reserved bit violations; they are cases when a ToPA entry is encountered with 
illegal field combinations. They include the following: 


a. Setting the STOP or INT bit on an entry with END=1. 

b. Setting the END bit in entry 0 of a ToPA table. 

c. On processors that support only a single ToPA entry (see above), two additional illegal settings apply: 
i) ToPA table entry 1 with END=O. 
ii) ToPA table entry 1 with base address not matching the table base. 


In all cases, the error will be logged by setting IA32_RTIT_STATUS.Error, thereby disabling tracing when the prob- 
lematic ToPA entry is reached (when proc_trace_table_offset points to the entry containing the error). Any packet 
bytes that are internally buffered when the error is detected may be lost. 


Note that operational errors may also be signaled due to attempts to access restricted memory. See Section 
31.2.6.4 for details. 


A tracing software have a range of flexibility using ToPA to manage the interaction of Intel PT with application 
buffers, see Section 31.4.2.26. 


31.2.6.3 Trace Transport Subsystem 


When IA32_RTIT_CTL.FabricEn is set, the IA32_RTIT_CTL.ToPA bit is ignored, and trace output is written to the 
trace transport subsystem. The endpoints of this transport are platform-specific, and details of configuration 
options should refer to the specific platform documentation. The FabricEn bit is available to be set if 
CPUID(EAX=14H,ECX=0):EBX[bit 3] = 1. 


31.2.6.4 Restricted Memory Access 


Packet output cannot be directed to any regions of memory that are restricted by the platform. In particular, all 
memory accesses on behalf of packet output are checked against the SMRR regions. If there is any overlap with 
these regions, trace data collection will not function properly. Exact processor behavior is implementation-depen- 
dent; Table 31-5 summarizes several scenarios. 


Table 31-5. Behavior on Restricted Memory Access 


Scenario Description 
ToPA output region | Stores to the restricted memory region will be dropped, and that packet data will be lost. Any attempt to read 
overlaps with from that restricted region will return all 1s. The processor also may signal an error (Section 31.3.9) and disable 
SMRR tracing when the output pointer reaches the restricted region. If packet generation remains enabled, then 


packet output may continue once stores are no longer directed to restricted memory (on wrap, or if the output 
region is larger than the restricted memory region). 


ToPA table overlaps | The processor will signal an error (Section 31.3.9) and disable tracing when the ToPA write pointer 
with SMRR (IA32_RTIT_OUTPUT_BASE + proc_trace_table_offset) enters the restricted region. 


It should also be noted that packet output should not be routed to the 4KB APIC MMIO region, as defined by the 
IA32_APIC_BASE MSR. For details about the APIC, refer to Intel® 64 and IA-32 Architectures Software Devel- 
oper’s Manual, Volume 3A. No error is signaled for this case. 


Modifications to Restricted Memory Regions 


It is recommended that software disable packet generation before modifying the SMRRs to change the scope of the 
SMRR regions. This is because the processor reserves the right to cache any number of ToPA table entries inter- 
nally, after checking them against restricted memory ranges. Once cached, the entries will not be checked again, 
meaning one could potentially route packet output to a newly restricted region. Software can ensure that any 
cached entries are written to memory by clearing IA32_RTIT_CTL.TraceEn. 
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31.2.7. Enabling and Configuration MSRs 


31.2.7.1 General Considerations 


Trace packet generation is enabled and configured by a collection of model-specific registers (MSRs), which are 
detailed below. Some notes on the configuration MSR behavior: 


If Intel Processor Trace is not supported by the processor (see Section 31.3.1), RDMSR or WRMSR of the 
IA32_RTIT_* MSRs will cause #GP. 


A WRMSR to any of these configuration MSRs that begins and ends with IA32_RTIT_CTL.TraceEn set will #GP 
fault. Packet generation must be disabled before the configuration MSRs can be changed. 


Note: Software may write the same value back to IA32_RTIT_CTL without #GP, even if TraceEn=1. 
All configuration MSRs for Intel PT are duplicated per logical processor 


For each configuration MSR, any MSR write that attempts to change bits marked reserved, or utilize encodings 
marked reserved, will cause a #GP fault. 


All configuration MSRs for Intel PT are cleared on a warm or cold RESET. 


— If CPUID.(EAX=14H, ECX=0):EBX[bit 2] = 1, only the TraceEn bit is cleared on warm RESET; though this 
may have the impact of clearing other bits in IA32_RTIT_STATUS. Other MSR values of the trace configu- 
ration MSRs are preserved on warm RESET. 


The semantics of MSR writes to trace configuration MSRs in this chapter generally apply to explicit WRMSR to 
these registers, using VM-exit or VM-entry MSR load list to these MSRs, XRSTORS with requested feature bit 
map including XSAVE map component of state_8 (corresponding to IA32_XSS[bit 8]), and the write to 
IA32_RTIT_CTL.TraceEn by XSAVES (Section 31.3.5.2). 


31.2.7.2 IA32_RTIT_CTL MSR 


IA32_RTIT_CTL, at address 570H, is the primary enable and control MSR for trace packet generation. Bit positions 
are listed in Table 31-6. 


Table 31-6. IA32_RTIT_CTL MSR 


Position | BitName | At Reset Bit Description 


0 TraceEn 0) If 1, enables tracing; else tracing is disabled. 


When this bit transitions from 1 to O, all buffered packets are flushed out of internal buffers. 
A further store, fence, or architecturally serializing instruction may be required to ensure that 
packet data can be observed at the trace endpoint. See Section 31.2.7.3 for details of 
enabling and disabling packet generation. 


Note that the processor will clear this bit on #SMI (Section 31.2.8.3) and warm reset. Other 
MSR bits of IA32_RTIT_CTL (and other trace configuration MSRs) are not impacted by these 
events. 


1 CYCEn 0 0: Disables CYC Packet (see Section 31.4.2.14). 
1: Enables CYC Packet. 
This bit is reserved if CPUID.(EAX=14H, ECX=0):EBX[bit 1] = 0. 


2 OS 0 0: Packet generation is disabled when CPL = 0. 
1: Packet generation may be enabled when CPL = 0. 


3 User 0 0: Packet generation is disabled when CPL > O. 
1: Packet generation may be enabled when CPL > O. 


4 Pwr€vtEn 0 O: Power Event Trace packets are disabled. 
1: Power Event Trace packets are enabled (see Section 31.2.3, “Power Event Tracing”). 
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Table 31-6. [A32_RTIT_CTL MSR (Contd.) 


Position | BitName_ | At Reset Bit Description 


5 FUPonPTW 0 0: PTW packets are not followed by FUPs. 
1: PTW packets are followed by FUPs. 
This bit is reserved when CPUID.(EAX=14H, ECX=0):EBX[bit 4] ("PTWRITE Supported”) is 0. 


6 FabricEn 0 0: Trace output is directed to the memory subsystem, mechanism depends on 
IA32_RTIT_CTL.ToPA. 

1: Trace output is directed to the trace transport subsystem, I[A32_RTIT_CTL.ToPA is ignored. 
This bit is reserved if CPUID.(EAX=14H, ECX=0):ECX[bit 3] = 0. 


7 CR3Filter 0 0: Disables CR3 filtering. 
1: Enables CR3 filtering. 
This bit is reserved if CPUID.(EAX=14H, ECX=0):EBX[bit 0] (“CR3 Filtering Support”) is 0. 


8 ToPA 0 0: Single-range output scheme enabled if CPUID.(EAX=14H, ECX=0):ECX.SNGLRGNOUT[bit 2] 
= 1 and IA32_RTIT_CTL.FabricEn=0. 

1: ToPA output scheme enabled (see Section 31.2.6.2) if CPUID.(EAX=14H, 
ECX=0):ECX.TOPA[bit 0] = 1, and I|A32_RTIT_CTL.FabricEn=0. 

Note: WRMSR to IA32_RTIT_CTL that sets TraceEn but clears this bit and FabricEn would 
cause #GP, if CPUID.(EAX=14H, ECX=0):ECX.SNGLRGNOUT[bit 2] = 0. 

WRMSR to IA32_RTIT_CTL that sets this bit causes #GP, if CPUID.(EAX=14H, 
ECX=0):ECX.TOPA[bit 0] = 0. 


9 MTCEn 0 0: Disables MTC Packet (see Section 31.4.2.16). 
1: Enables MTC Packet. 
This bit is reserved if CPUID(EAX=14H, ECX=0):EBX[bit 3] = 0. 


10 TSCEn 0 0: Disable TSC packets. 

1: Enable TSC packets (see Section 31.4.2.11). 
11 DisRETC 0 0: Enable RET compression. 

1: Disable RET compression (see Section 31.2.1.2). 
12 PTWEn 0 0: PTWRITE packet generation disabled. 


1: PTWRITE packet generation enabled (see Table 31-41 “PTW Packet Definition’). 
This bit is reserved when CPUID.(EAX=14H, ECX=0):EBX[bit 4] ("PTWRITE Supported”) is 0. 


13 Branch€n 0 0: Disable COFI-based packets. 
1: Enable COFI-based packets: FUP, TIP, TIP.PGE, TIP.PGD, TNT, MODE.Exec, MODE.TSX. 
See Section 31.2.5.4 for details on Branch€n. 


17:14 | MTCFreq 0 Defines MTC packet Frequency, which is based on the core crystal clock, or Always Running 
Timer (ART). MTC will be sent each time the selected ART bit toggles. The following Encodings 
are defined: 

0: ART(0), 1: ART(1), 2: ART(2), 3: ART(3), 4: ART(4), 5: ART(5), 6: ART(6), 7: ART(7), 
8: ART(8), 9: ART(9), 10: ART(10), 11: ART(11), 12: ART(12), 13: ART(13), 14: ART(14), 15: 

ART(15) 

Software must use CPUID to query the supported encodings in the processor, see Section 
31.3.1. Use of unsupported encodings will result in a #GP fault. This field is reserved if 
CPUID.(EAX=1 4H, ECX=0):EBX[bit 3] = 0. 


18 Reserved 0 Must be 0. 
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Table 31-6. IA32_RTIT_CTL MSR (Contd.) 


Position BitName_ | At Reset Bit Description 


22:19 | CycThresh 0 CYC packet threshold, see Section 31.3.6 for details. CYC packets will be sent with the first 
eligible packet after N cycles have passed since the last CYC packet. If CycThresh is O then 
N=O, otherwise N is defined as 2(CY*"hresh-1) The following Encodings are defined: 

0: 0, 1: 1, 2:2, 3:4, 4:8, 5: 16, 6: 32, 7: 64, 
8:128,9:256, 10:512, 11:1024, 12:2048, 13:4096, 14:8192, 15:16384 
Software must use CPUID to query the supported encodings in the processor, see Section 
31.3.1. Use of unsupported encodings will result in a #GP fault. This field is reserved if 
CPUID.(EAX=14H, ECX=0):EBX[bit 1] = 0. 


23 Reserved 0 Must be 0. 


27:24 | PSBFreq 0 Indicates the frequency of PSB packets. PSB packet frequency is based on the number of Intel 
PT packet bytes output, so this field allows the user to determine the increment of 
IA32_IA32_RTIT_STATUS.PacketByteCnt that should cause a PSB to be generated. Note that 
PSB insertion is not precise, but the average output bytes per PSB should approximate the 
SW selected period. The following Encodings are defined: 

0: 2K, 1: 4K, 2: BK, 3: 16K, 4: 32K, 5: 64K, 6:128K, 7: 256K, 
8:512K,9:1M, 10: 2M, 11: 4M, 12: 8M, 13:16M, 14:32M, 15:64M 
Software must use CPUID to query the supported encodings in the processor, see Section 
31.3.1. Use of unsupported encodings will result in a #GP fault. This field is reserved if 
CPUID.(EAX=14H, ECX=0):EBX[bit 1] = 0. 


31:28 | Reserved 0 Must be 0. 
35:32 | ADDRO_CFG |0 Configures the base/limit register pair |A32_RTIT_ADDRO_A/B based on the following 
encodings: 


0: ADDRO range unused. 


1: The [I[A32_RTIT_ADDRO_A..IA32_RTIT_ADDRO_B] range defines a FilterEn range. FilterEn 
will only be set when the IP is within this range, though other FilterEn ranges can additionally 
be used. See Section 31.2.4.3 for details on IP filtering. 


2: The [IA32_RTIT_ADDRO_A..IA32_RTIT_ADDRO_B] range defines a TraceStop range. 
TraceStop will be asserted if code branches into this range. See 4.2.8 for details on TraceStop. 


3..15: Reserved (#GP). 
This field is reserved if CPUID.(EAX=14H, ECX=1):EBX.RANGECNT[2:0] < 1. 


39:36 | ADDR1I_CFG |0 Configures the base/limit register pair [A32_RTIT_ADDR1_A/B based on the following 
encodings: 

0: ADDR1 range unused. 

1: The [IA32_RTIT_ADDR1_A..IA32_RTIT_ADDR1_B] range defines a FilterEn range. FilterEn 
will only be set when the IP is within this range, though other FilterEn ranges can additionally 
be used. See Section 31.2.4.3 for details on IP filtering. 


2: The [IA32_RTIT_ADDR1_A.1A32_RTIT_ADDR1_B] range defines a TraceStop range. 
TraceStop will be asserted if code branches into this range. See Section 31.4.2.10 for details 
on TraceStop. 

3..15: Reserved (#GP). 


This field is reserved if CPUID(EAX=14H, ECX=1):EBX.RANGECNT[2:0] < 2. 
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Table 31-6. IA32_RTIT_CTL MSR (Contd.) 
Position | BitName_ | At Reset Bit Description 
43:40 | ADDR2_CFG |0 Configures the base/limit register pair |A32_RTIT_ADDR2_A/B based on the following 
encodings: 
0: ADDR2 range unused. 
1: The [IA32_RTIT_ADDR2_A..IA32_RTIT_ADDR2_B] range defines a FilterEn range. FilterEn 


will only be set when the IP is within this range, though other FilterEn ranges can additionally 
be used. See Section 31.2.4.3 for details on IP filtering. 


2: The [IA32_RTIT_ADDR2_A..|A32_RTIT_ADDR2_B] range defines a TraceStop range. 
TraceStop will be asserted if code branches into this range. See Section 31.4.2.10 for details 
on TraceStop. 

3..15: Reserved (#GP). 

This field is reserved if CPUID.(EAX=14H, ECX=1):EBX.RANGECNT[2:0] < 3. 

47:44 | ADDR3_CFG |0 Configures the base/limit register pair |A32_RTIT_ADDR3_A/B based on the following 
encodings: 

0: ADDR3 range unused. 

1: The [IA32_RTIT_ADDR3_A..IA32_RTIT_ADDR3_B] range defines a FilterEn range. FilterEn 


will only be set when the IP is within this range, though other FilterEn ranges can additionally 
be used. See Section 31.2.4.3 for details on IP filtering. 


2: The [IA32_RTIT_ADDR3_A..IA32_RTIT_ADDR3_B] range defines a TraceStop range. 
TraceStop will be asserted if code branches into this range. See Section 31.4.2.10 for details 
on TraceStop. 


3..15: Reserved (#GP). 
This field is reserved if CPUID.(EAX=14H, ECX=1):EBX.RANGECNT[2:0] < 4. 


55:48 | Reserved 0 Reserved only for future trace content enables, or address filtering configuration enables. 
Must be 0. 
56 InjectPsbPmi | 0 1: Enables use of IA32_RTIT_STATUS bits PendPSB[6] and PendTopaPMI[7], see Section 
On€Enable 31.2.7.4, “IA32_RTIT_STATUS MSR" for behavior of these bits. 


0: IA32_RTIT_STATUS bits 6 and 7 are ignored. 
This field is reserved if CPUID.(EAX=14H, ECX=0):EBX[bit 6] = 0. 


59:57 | Reserved 0 Reserved only for future trace content enables, or address filtering configuration enables. 
Must be 0. 
63:60 | Reserved 0 Must be 0. 


31.2.7.3. Enabling and Disabling Packet Generation with TraceEn 


When TraceEn transitions from 0 to 1, Intel Processor Trace is enabled, and a series of packets may be generated. 
These packets help ensure that the decoder is aware of the state of the processor when the trace begins, and that 
it can keep track of any timing or state changes that may have occurred while packet generation was disabled. A 
full PSB+ (see Section 31.4.2.17) will be generated if IA32_RTIT_STATUS.PacketByteCnt=0, and may be gener- 
ated in other cases as well. Otherwise, timing packets will be generated, including TSC, TMA, and CBR (see Section 
31.4.1.1). 


In addition to the packets discussed above, if and when PacketEn (Section 31.2.5.1) transitions from 0 to 1 (which 
may happen immediately, depending on filtering settings), a TIP.PGE packet (Section 31.4.2.3) will be generated. 


When TraceEn is set, the processor may read ToPA entries from memory and cache them internally. For this reason, 
software should disable packet generation before making modifications to the ToPA tables (or changing the config- 
uration of restricted memory regions). See Section 31.7 for more details of packets that may be generated with 
modifications to TraceEn. 
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Disabling Packet Generation 


Clearing TraceEn causes any packet data buffered within the logical processor to be flushed out, after which the 
output MSRs (IA32_RTIT_OUTPUT_BASE and IA32_RTIT_OUTPUT_MASK_PTRS) will have stable values. When 
output is directed to memory, a store, fence, or architecturally serializing instruction may be required to ensure 
that the packet data is globally observed. No special packets are generated by disabling packet generation, though 
a TIP.PGD may result if PacketEn=1 at the time of disable. 


Other Writes to IA32_RTIT_CTL 


Any attempt to modify IA32_RTIT_CTL while TraceEn is set will result in a general-protection fault (#GP) unless the 
same write also clears TraceEn. However, writes to IA32_RTIT_CTL that do not modify any bits will not cause a #GP, 
even if TraceEn remains set. 


31.2.7.4 IA32_RTIT_STATUS MSR 


The IA32_RTIT_STATUS MSR is readable and writable by software, though some fields cannot be modified by soft- 
ware. See Table 31-7 for details. The WRMSR instruction ignores these bits in the source operand (attempts to 
modify these bits are ignored and do not cause WRMSR to fault). 


This MSR can only be written when IA32_RTIT_CTL.TraceEn is 0; otherwise WRMSR causes a general-protection 
fault (#GP). The processor does not modify the value of this MSR while TraceEn is 0 (software can modify it with 
WRMSR). 


Table 31-7. |A32_RTIT_STATUS MSR 


Position Bit Name At Reset Bit Description 

0 FilterEn 0 This bit is written by the processor, and indicates that tracing is allowed for the current IP, 
see Section 31.2.5.5. Writes are ignored. 

1 ContextEn 0 The processor sets this bit to indicate that tracing is allowed for the current context. See 
Section 31.2.5.3. Writes are ignored. 

2 TriggerEn 0 The processor sets this bit to indicate that tracing is enabled. See Section 31.2.5.2. Writes are 
ignored. 

3 Reserved 0 Must be 0. 

4 Error 0 The processor sets this bit to indicate that an operational error has been encountered. When 


this bit is set, TriggerEn is cleared to O and packet generation is disabled. For details, see 
“ToPA Errors” in Section 31.2.6.2. 

When Trace€En is cleared, software can write this bit. Once it is set, only software can clear it. 
It is not recommended that software ever set this bit, except in cases where it is restoring a 
prior saved state. 


5 Stopped 0 The processor sets this bit to indicate that a ToPA Stop condition has been encountered. 
When this bit is set, TriggerEn is cleared to 0 and packet generation is disabled. For details, 
see “ToPA STOP" in Section 31.2.6.2. 

When Trace€n is cleared, software can write this bit. Once it is set, only software can clear it. 
It is not recommended that software ever set this bit, except in cases where it is restoring a 
prior saved state. 


6 PendPSB 0 If IA32_RTIT_CTL.InjectPsbPmiOnEnable[56] = 1, the processor sets this bit when the 
threshold for a PSB+ to be inserted has been reached. The processor will clear this bit when 
the PSB+ has been inserted into the trace. If PendPSB = 1 and InjectPsbPmiOnEnable = 1 
when IA32_RTIT_CTL.TraceEn[0] transitions from 0 to 1, a PSB+ will be inserted into the 
trace. 


This field is reserved if CPUID.(EAX=14H, ECX=0):EBX[bit 6] = 0. 
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Table 31-7. IA32_RTIT_STATUS MSR 
Position Bit Name At Reset Bit Description 


7 PendTopaPMI | 0 If IA32_RTIT_CTL.InjectPsbPmiOnEnable[56] = 1, the processor sets this bit when the 
threshold for a ToPA PMI to be inserted has been reached. Software should clear this bit once 
the ToPA PMI has been handled, see “ToPA PMI” for details. lf PendTopaPMI = 1 and 
InjectPsbPmiOnEnable = 1 when IA32_RTIT_CTL.TraceEn[0] transitions from 0 to 1, a PMI will 
be pended. 


This field is reserved if CPUID.(EAX=14H, ECX=0):EBX[bit 6] = 0. 
Must be 0. 


This field is written by the processor, and holds a count of packet bytes that have been sent 
out. The processor also uses this field to determine when the next PSB packet should be 
inserted. Note that the processor may clear or modify this field at any time while 
IA32_RTIT_CTL.TraceEn=1. It will have a stable value when IA32_RTIT_CTL.TraceEn=0. 


See Section 31.4.2.17 for details. 


This field is reserved when CPUID.(EAX=14H,ECX=0):EBX[bit 1] (“Configurable PSB and 
CycleAccurate Mode Supported") is 0. 


63:49 | Reserved 0 Must be 0. 


31:8 Reserved 
48:32 | PacketByteCnt 


jo) 


(=) 


31.2.7.5 1A32_RTIT_ADDRn_A and IA32_RTIT_ADDRn_B MSRs 


The role of the IA32_RTIT_ADDRn_A/B register pairs, for each n, is determined by the corresponding ADDRn_CFG 
fields in IA32_RTIT_CTL (see Section 31.2.7.2). The number of these register pairs is enumerated by 
CPUID.(EAX=14H, ECX=1):EAX.RANGECNT[2:0]. 


° Processors that enumerate support for 1 range support: 
IA32_RTIT_ADDRO_A, IA32_RTIT_ADDRO_B 

¢ Processors that enumerate support for 2 ranges support: 
IA32_RTIT_ADDRO_A, IA32_RTIT_ADDRO_B 
IA32_RTIT_ADDR1_A, IA32_RTIT_ADDR1_B 

¢ Processors that enumerate support for 3 ranges support: 
IA32_RTIT_ADDRO_A, IA32_RTIT_ADDRO_B 
IA32_RTIT_ADDR1_A, IA32_RTIT_ADDR1_B 
IA32_RTIT_ADDR2_A, IA32_RTIT_ADDR2_B 

¢ Processors that enumerate support for 4 ranges support: 
IA32_RTIT_ADDRO_A, IA32_RTIT_ADDRO_B 
IA32_RTIT_ADDR1_A, IA32_RTIT_ADDR1_B 
IA32_RTIT_ADDR2_A, IA32_RTIT_ADDR2_B 
IA32_RTIT_ADDR3_A, IA32_RTIT_ADDR3_B 


Each register has a single 64-bit field that holds a linear address value. Writes must ensure that the address is in 
canonical form, otherwise a general-protection fault (#GP) fault will result. 


Each MSR can be written only when IA32_RTIT_CTL.TraceEn is 0; otherwise WRMSR causes a general-protection 
fault (#GP). 


31.2.7. 1IA32_RTIT_CR3_MATCH MSR 


The IA32_RTIT_CR3_MATCH register is compared against CR3 when IA32_RTIT_CTL.CR3Filter is 1. Bits 63:5 hold 
the CR3 address value to match, bits 4:0 are reserved to 0. For more details on CR3 filtering and the treatment of 
this register, see Section 31.2.4.2. 


This MSR is accessible if CPUID.(EAX=14H, ECX=0):EBX[bit 0], “CR3 Filtering Support”, is 1. This MSR can be 
written only when IA32_RTIT_CTL.TraceEn is 0; otherwise WRMSR causes a general-protection fault (#GP). 
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IA32_RTIT_CR3_MATCH[4:0] are reserved and must be 0; an attempt to set those bits using WRMSR causes a 
#GP. 


31.2.7.7. IA32_RTIT_OUTPUT_BASE MSR 


This MSR is used to configure the trace output destination, when output is directed to memory 
(IA32_RTIT_CTL.FabricEn = 0). The size of the address field is determined by the maximum physical address width 
(MAXPHYADDR), as reported by CPUID.80000008H:EAX[7:0]. 


When the ToPA output scheme is used, the processor may update this MSR when packet generation is enabled, and 
those updates are asynchronous to instruction execution. Therefore, the values in this MSR should be considered 
unreliable unless packet generation is disabled (IA32_RTIT_CTL.TraceEn = 0). 


Accesses to this MSR are supported only if Intel PT output to memory is supported, hence when either 
CPUID.(EAX=14H, ECX=0):ECX[bit 0] or CPUID.(EAX=14H, ECX=0):ECX[bit 2] are set. Otherwise WRMSR or 
RDMSR cause a general-protection fault (4#GP). If supported, this MSR can be written only when 
IA32_RTIT_CTL.TraceEn is 0; otherwise WRMSR causes a general-protection fault (#GP). 


Table 31-8. |A32_RTIT_OUTPUT_BASE MSR 


Position Bit Name At Reset Bit Description 
6:0 Reserved 0 Must be 0. 
MAXPHYADDR-1:7 | BasePhysAddr | 0 The base physical address. How this address is used depends on the value of 


IA32_RTIT_CTL.ToPA: 


0: This is the base physical address of a single, contiguous physical output region. 
This could be mapped to DRAM or to MMIO, depending on the value. 


The base address should be aligned with the size of the region, such that none of 
the 1s in the mask value(Section 31.2.7.8) overlap with 1s in the base address. If 
the base is not aligned, an operational error will result (see Section 31.3.9). 


1: The base physical address of the current ToPA table. The address must be 4K 
aligned. Writing an address in which bits 11:7 are non-zero will not cause a #GP, but 
an operational error will be signaled once Trace€En is set. See “ToPA Errors” in 
Section 31.2.6.2 as well as Section 31.3.9. 


63:MAXPHYADDR | Reserved 0 Must be 0. 


31.2.7.8  IA32_RTIT_.OUTPUT_MASK_PTRS MSR 


This MSR holds any mask or pointer values needed to indicate where the next byte of trace output should be 
written. The meaning of the values held in this MSR depend on whether the ToPA output mechanism is in use. See 
Section 31.2.6.2 for details. 


The processor updates this MSR while when packet generation is enabled, and those updates are asynchronous to 
instruction execution. Therefore, the values in this MSR should be considered unreliable unless packet generation 
is disabled (IA32_RTIT_CTL.TraceEn = 0). 


Accesses to this MSR are supported only if Intel PT output to memory is supported, hence when either 
CPUID.(EAX=14H, ECX=0):ECX[bit 0] or CPUID.(EAX=14H, ECX=0):ECX[bit 2] are set. Otherwise WRMSR or 
RDMSR cause a general-protection fault (#GP). If supported, this MSR can be written only when 
IA32_RTIT_CTL.TraceEn is 0; otherwise WRMSR causes a general-protection fault (#GP). 
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Table 31-9. IA32_RTIT_OUTPUT_MASK_PTRS MSR 


Position Bit Name At Reset Bit Description 
6:0 LowerMask 7FH Forced to 1, writes are ignored. 
31:7 MaskOrTableO | 0 The use of this field depends on the value of [A32_RTIT_CTL.ToPA: 
ffset 0: This field holds bits 31:7 of the mask value for the single, contiguous physical output 


region. The size of this field indicates that regions can be of size 128B up to 4GB. This value 
(combined with the lower 7 bits, which are reserved to 1) will be ANDed with the 
OutputOffset field to determine the next write address. All 1s in this field should be 
consecutive and starting at bit 7, otherwise the region will not be contiguous, and an 
operational error (Section 31.3.9) will be signaled when TraceEn is set. 


1: This field holds bits 27:3 of the offset pointer into the current ToPA table. This value can 
be added to the [A32_RTIT_OUTPUT_BASE value to produce a pointer to the current ToPA 
table entry, which itself is a pointer to the current output region. In this scenario, the lower 7 
reserved bits are ignored. This field supports tables up to 256 MBytes in size. 


63:32 | OutputOffset | O The use of this field depends on the value of [A32_RTIT_CTL.ToPA: 


0: This is bits 31:0 of the offset pointer into the single, contiguous physical output region. 
This value will be added to the [A32_RTIT_OUTPUT_BASE value to form the physical address 
at which the next byte of packet output data will be written. This value must be less than or 
equal to the MaskOrTableOffset field, otherwise an operational error (Section 31.3.9) will be 
signaled when Trace €n is set. 

1: This field holds bits 31:0 of the offset pointer into the current ToPA output region. This 
value will be added to the output region base field, found in the current ToPA table entry, to 
form the physical address at which the next byte of trace output data will be written. 

This value must be less than the ToPA entry size, otherwise an operational error (Section 
31.3.9) will be signaled when Trace€En is set. 


31.2.8 Interaction of Intel® Processor Trace and Other Processor Features 


31.2.8.1 Intel® Transactional Synchronization Extensions (Intel® TSX) 


The operation of Intel TSX is described in Chapter 14 of the Inte/® 64 and IA-32 Architectures Software Devel- 
oper’s Manual, Volume 1. For tracing purpose, packet generation does not distinguish between hardware lock 
elision (HLE) and restricted transactional memory (RTM), but speculative execution does have impacts on the trace 
output. Specifically, packets are generated as instructions complete, even for instructions in a transactional region 
that is later aborted. For this reason, debugging software will need indication of the beginning and end of a trans- 
actional region; this will allow software to understand when instructions are part of a transactional region and 
whether that region has been committed. 

To enable this, TSX information is included in a MODE packet leaf. The mode bits in the leaf are: 

¢ InTX: Set to 1 on an TSX transaction begin, and cleared on transaction commit or abort. 

¢ TXAbort: Set to 1 only when InTX transitions from 1 to 0 on an abort. Cleared otherwise. 

If BranchEn=1, this MODE packet will be sent each time the transaction status changes. See Table 31-10 for 
details. 


Table 31-10. TSX Packet Scenarios 
TSX Event Instruction Packets 


Transaction Begin | Either XBEGIN or XACQUIRE lock (the latter if executed MODE(TXAbort=0, InTX=1), FUP(CurrentIP) 
transactionally) 


Transaction Either XEND or XRELEASE lock, if transactional execution MODE(TXAbort=0, InTX=0), FUP(CurrentIP) 
Commit ends. This happens only on the outermost commit 
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Table 31-10. TSX Packet Scenarios 


TSX Event Instruction Packets 
Transaction Abort | XABORT or other transactional abort MODE(TXAbort=1, InTX=0), FUP(CurrentlP), 
TIP(TargetIP) 
Other One of the following: None. No change to TSX mode bits for these 

« Nested XBEGIN or XACQUIRE lock cases. 

* Anouter XACQUIRE lock that doesn’t begin a transaction 

(INTX not set) 
* Non-outermost XEND or XRELEASE lock 


The CurrentIP listed above is the IP of the associated instruction. The TargetIP is the IP of the next instruction to be 
executed; for HLE, this is the XACQUIRE lock; for RTM, this is the fallback handler. 


Intel PT stores are non-transactional, and thus packet writes are not rolled back on TSX abort. 


31.2.8.2 TSX and IP Filtering 


A complication with tracking transactions is handling transactions that start or end outside of the tracing region. 
Transactions can’t span across a change in ContextEn, because CPL changes and CR3 changes each cause aborts. 
But a transaction can start within the IP filter region and end outside it. 


To assist the decoder handling this situation, MODE.TSX packets can be sent even if FilterEn=0, though there will 
be no FUP attached. Instead, they will merely serve to indicate to the decoder when transactions are active and 
when they are not. When tracing resumes (due to PacketEn=1), the last MODE.TSX preceding the TIP.PGE will indi- 
cate the current transaction status. 


31.2.8.3 System Management Mode (SMM) 


SMM code has special privileges that non-SMM code does not have. Intel Processor Trace can be used to trace SMM 
code, but special care is taken to ensure that SMM handler context is not exposed in any non-SMM trace collection. 
Additionally, packet output from tracing non-SMM code cannot be written into memory space that is either 
protected by SMRR or used by the SMM handler. 


SMM is entered via a system management interrupt (SMI). SMI delivery saves the value of IA32_RTIT_CTL.TraceEn 
into SMRAM and then clears it, thereby disabling packet generation. 


The saving and clearing of IA32_RTIT_CTL.TraceEn ensures two things: 
1. All internally buffered packet data is flushed before entering SMM (see Section 31.2.7.2). 


2. Packet generation ceases before entering SMM, so any tracing that was configured outside SMM does not 
continue into SMM. No SMM instruction pointers or other state will be exposed in the non-SMM trace. 


When the RSM instruction is executed to return from SMM, the TraceEn value that was saved by SMI delivery is 
restored, allowing tracing to be resumed. As is done any time packet generation is enabled, ContextEn is re-evalu- 
ated, based on the values of CPL, CR3, etc., established by RSM. 


Like other interrupts, delivery of an SMI produces a FUP containing the IP of the next instruction to execute. By 
toggling TraceEn, SMI and RSM can produce TIP.PGD and TIP.PGE packets, respectively, indicating that tracing was 
disabled or re-enabled. See Table 31.7 for more information about packets entering and leaving SMM. 


Although #SMI and RSM change CR3, PIP packets are not generated in these cases. With #SMI tracing is disabled 
before the CR3 change; with RSM TraceEn is restored after CR3 is written. 


TraceEn must be cleared before executing RSM, otherwise it will cause a shutdown. Further, on processors that 
restrict use of Intel PT with LBRs (see Section 31.3.1.2), any RSM that results in enabling of both will cause a shut- 
down. 


Intel PT can support tracing of System Transfer Monitor operating in SMM, see Section 31.6. 
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31.2.8.4 Virtual-Machine Extensions (VMX) 


Initial implementations of Intel Processor Trace do not support tracing in VMX operation. Such processors indicate 
this by returning O for IA32_VMX_MISC[bit 14]. On these processors, execution of the VMXON instruction clears 
IA32_RTIT_CTL.TraceEn and any attempt to write IA32_RTIT_CTL in VMX operation causes a general-protection 
exception (#GP). 

Processors that support Intel Processor Trace in VMX operation return 1 for IA32_VMX_MISC[bit 14]. Details of 
tracing in VMX operation are described in Section 31.4.2.26. 


31.2.8.5 Intel® Software Guard Extensions (Intel® SGX) 


Intel SGX provides an application with the ability to instantiate a protective container (an enclave) with confidenti- 
ality and integrity (see the Inte/® Software Guard Extensions Programming Reference). On a processor with both 
Intel PT and Intel SGX enabled, when executing code within a production enclave, no control flow packets are 
produced by Intel PT. An enclave entry will clear ContextEn, thereby blocking control flow packet generation. A 
TIP.PGD packet will be generated if PacketEn=1 at the time of the entry. 


Upon enclave exit, ContextEn will no longer be forced to 0. If other enables are set at the time, a TIP.PGE may be 
generated to indicate that tracing is resumed. 

During the enclave execution, Intel PT remains enabled, and periodic or timing packets such as PSB, TSC, MTC, or 
CBR can still be generated. No IPs or other architectural state will be exposed. 


For packet generation examples on enclave entry or exit, see Section 31.7. 


Debug Enclaves 


Intel SGX allows an enclave to be configured with relaxed protection of confidentiality for debug purposes, see the 
Intel® Software Guard Extensions Programming Reference. In a debug enclave, Intel PT continues to function 
normally. Specifically, ContextEn is not impacted by an enclave entry or exit. Hence, the generation of ContextEn- 
dependent packets within a debug enclave is allowed. 


31.2.8.6 SENTER/ENTERACCS and ACM 


GETSEC[SENTER] and GETSEC[ENTERACCS] instructions clear TraceEn, and it is not restored when those instruc- 
tion complete. SENTER also causes TraceEn to be cleared on other logical processors when they rendezvous and 
enter the SENTER sleep state. In these two cases, the disabling of packet generation is not guaranteed to flush 
internally buffered packets. Some packets may be dropped. 

When executing an authenticated code module (ACM), packet generation is silently disabled during ACRAM setup. 
TraceEn will be cleared, but no TIP.PGD packet is generated. After completion of the module, the TraceEn value will 
be restored. There will be no TIP.PGE packet, but timing packets, like TSC and CBR, may be produced. 


31.2.8.7. Intel® Memory Protection Extensions (Intel? MPX) 


Bounds exceptions (#BR) caused by Intel MPX are treated like other exceptions, producing FUP and TIP packets 
that indicate the source and destination IPs. 


31.3. CONFIGURATION AND PROGRAMMING GUIDELINE 


31.3.1. Detection of Intel Processor Trace and Capability Enumeration 


Processor support for Intel Processor Trace is indicated by CPUID.(EAX=07H, ECX=0H):EBX[bit 25] = 1. CPUID 
function 14H is dedicated to enumerate the resource and capability of processors that report 
CPUID.(EAX=07H,ECX=0H):EBX[bit 25] = 1. Different processor generations may have architecturally-defined 
variation in capabilities. Table 31-11 describes details of the enumerable capabilities that software must use across 
generations of processors that support Intel Processor Trace. 
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Table 31-11. CPUID Leaf 14H Enumeration of Intel Processor Trace Capabilities 


CPUID.(EAX=14H,ECX=0) 


Register 


Bits 


Name 


Description Behavior 


EAX 


31:0 


Maximum valid sub-leaf Index 


Specifies the index of the maximum valid sub-leaf for this CPUID leaf. 


EBX 


0 


CR3 Filtering Support 


1: Indicates that IA32_RTIT_CTL.CR3Filter can be set to 1, and that 
IA32_RTIT_CR3_MATCH MSR can be accessed. See Section 31.2.7. 

O: Indicates that writes that set |A32_RTIT_CTL.CR3Filter to 1, or any 
access to [A32_RTIT_CR3_MATCH, will #GP fault. 


Configurable PSB and Cycle- 
Accurate Mode Supported 


IP Filtering and TraceStop 
supported, and Preserve Intel 
PT MSRs across warm reset 


1: (a) |A32_RTIT_CTL.PSBFreq can be set to a non-zero value, in order to 
select the preferred PSB frequency (see below for allowed values). (b) 
IA32_RTIT_STATUS.PacketByteCnt can be set to a non-zero value, and 
will be incremented by the processor when tracing to indicate progress 
towards the next PSB. If trace packet generation is enabled by setting 
TraceEn, a PSB will only be generated if PacketByteCnt=0. (c) 
IA32_RTIT_CTL.CYCEn can be set to 1 to enable Cycle-Accurate Mode. 
See Section 31.2.7. 


QO: (a) Any attempt to write a non-zero value to |A32_RTIT_CTL.PSBFreq 
or IA32_RTIT_STATUS.PacketByteCnt will #GP fault. (b) If trace packet 
generation is enabled by setting TraceEn, a PSB is always generated. (c) 
Any attempt to write a non-zero value to [A32_RTIT_CTL.CYCEn or 
IA32_RTIT_CTL.CycThresh will #GP fault. 


1: (a) IA32_RTIT_CTL provides at one or more ADDRn_CFG field to 
configure the corresponding address range MSRs for IP Filtering or IP 
TraceStop. Each ADDRn_CFG field accepts a value in the range of 0:2 
inclusive. The number of ADDRn_CFG fields is reported by 
CPUID.(EAX=14H, ECX=1):EAX.RANGECNT[2:0]. (b) At least one register 
pair [A32_RTIT_ADDRn_A and IA32_RTIT_ADDRn_B are provided to 
configure address ranges for IP filtering or IP TraceStop. (c) On warm 
reset, all Intel PT MSRs will retain their pre-reset values, though 
IA32_RTIT_CTL.TraceEn will be cleared. The Intel PT MSRs are listed in 
Section 31.2.7. 

0: (a) An Attempt to write [A32_RTIT_CTLADDRn_CFG with non-zero 
encoding values will cause #GP. (b) Any access to IA32_RTIT_ADDRn_A 
and IA32_RTIT_ADDRn_B, will #GP fault. (c) On warm reset, all Intel PT 
MSRs will be cleared. 


MTC Supported 


1: IA32_RTIT_CTL.MTCEn can be set to 1, and MTC packets will be 
generated. See Section 31.2.7. 

0: An attempt to set IA32_RTIT_CTL.MTCEn or IA32_RTIT_CTL.MTCFreq 
to a non-zero value will #GP fault. 


PTWRITE Supported 


1: Writes can set [A32_RTIT_CTL[12] (PTWEn) and IA32_RTIT_CTL[5] 
(FUPonPTW), and PTWRITE can generate packets. 


0: Writes that set IA32_RTIT_CTL[1 2] or IA32_RTIT_CTL[5] will #GP, 
and PTWRITE will #UD fault. 


Power Event Trace Supported 


1: Writes can set IA32_RTIT_CTL[4] (Pwr€vtEn), enabling Power Event 
Trace packet generation. 


0: Writes that set IA32_RTIT_CTL[4] will #GP. 
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Table 31-11. CPUID Leaf 14H Enumeration of Intel Processor Trace Capabilities (Contd.) 


CPUID.(EAX=14H,ECX=0) 


Register Bits 


Name 


Description Behavior 


PSB and PMI Preservation 
Supported 


1: Writes can set [A32_RTIT_CTL[56] (InjectPsbPmiOnEnable), enabling 
the processor to set [A32_RTIT_STATUS[7] (PendTopaPMI) and/or 
IA32_RTIT_STATUS[6] (PendPSB) in order to preserve ToPA PMs and/or 
PSBs otherwise lost due to Intel PT disable. Writes can also set 
PendToPAPMI and PendPSB. 


0: Writes that set [A32_RTIT_CTL[56], |A32_RTIT_STATUS[7], or 
IA32_RTIT_STATUS[6] will #GP. 


Reserved 


ECX 


ToPA Output Supported 


1: Tracing can be enabled with [A32_RTIT_CTL.ToPA = 1, hence utilizing 
the ToPA output scheme (Section 31.2.6.2) IA32_RTIT_OUTPUT_BASE 
and |A32_RTIT_OUTPUT_MASK_PTRS MSRs can be accessed. 


0: Unless CPUID.(EAX=14H, ECX=0):ECX.SNGLRNGOUT[bit 2] = 1. writes 


to IA32_RTIT_OUTPUT_BASE or IA32_RTIT_OUTPUT_MASK_PTRS. 
MSRs will #GP fault. 


ToPA Tables Allow Multiple 
Output Entries 


1: ToPA tables can hold any number of output entries, up to the 
maximum allowed by the MaskOrTableOffset field of 
IA32_RTIT_OUTPUT_MASK_PTRS. 


0: ToPA tables can hold only one output entry, which must be followed 
by an END=1 entry which points back to the base of the table. 

Further, ToPA PMls will be delivered before the region is filled. See ToPA 
PMI in Section 31.2.6.2. 

If there is more than one output entry before the END entry, or if the 
END entry has the wrong base address, an operational error will be 
signaled (see “ToPA Errors” in Section 31.2.6.2). 


Single-Range Output 
Supported 


1: Enabling tracing (TraceEn=1) with IA32_RTIT_CTL.ToPA=0 is 
supported. 

0: Unless CPUID.(EAX=14H, ECX=0):ECX.TOPAOUT [bit 0] = 1. writes to 
IA32_RTIT_OUTPUT_BASE or IA32_RTIT_OUTPUT_MASK_PTRS. MSRs 
will #GP fault. 


Output to Trace Transport 
Subsystem Supported 


1: Setting IA32_RTIT_CTL.FabricEn to 1 is supported. 


0: |A32_RTIT_CTL.FabricEn is reserved. Write 1 to 
IA32_RTIT_CTLFabricEn will #GP fault. 


30:4 


Reserved 


EDX 31:0 


IP Payloads are LIP 


Reserved 


1: Generated packets which contain IP payloads have LIP values, which 
include the CS base component. 


0: Generated packets which contain IP payloads have RIP values, which 
are the offset from CS base. 


If CPUID.(EAX=14H, ECX=0):EAX reports a non-zero value, additional capabilities of Intel Processor Trace are 
described in the sub-leaves of CPUID leaf 14H. 
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Table 31-12. CPUID Leaf 14H, sub-leaf 1H Enumeration of Intel Processor Trace Capabilities 


CPUID.(EAX=14H,ECX=1) 


Register 


Bits 


Name 


Description Behavior 


EAX 


2:0 


Number of Address Ranges 


A non-zero value specifies the number ADDRn_CFG field supported in 
IA32_RTIT_CTL and the number of register pair 
IA32_RTIT_ADDRn_A/IA32_RTIT_ADDRn_B supported for IP filtering 
and IP TraceStop. 


NOTE: Currently, no processors support more than 4 address ranges. 


15:3 


Reserved 


31:16 


Bitmap of supported MTC 
Period Encodings 


The non-zero bits indicate the map of supported encoding values for 
the IA32_RTIT_CTL.MTCFreq field. This applies only if 
CPUID.(EAX=14H, ECX=0):EBX[bit 3] = 1 (MTC Packet generation is 
supported), otherwise the MTCFreq field is reserved to 0. 


Each bit position in this field represents 1 encoding value in the 4-bit 
MTCFreq field (ie, bit O is associated with encoding value 0). For each 
bit: 

1: MTCFreq can be assigned the associated encoding value. 

0: MTCFreq cannot be assigned to the associated encoding value. A 


write to [A32_RTIT_CTLMTCFreq with unsupported encoding will cause 
#GP fault. 


EBX 


Bitmap of supported Cycle 
Threshold values 


The non-zero bits indicate the map of supported encoding values for 
the IA32_RTIT_CTL.CycThresh field. This applies only if 
CPUID.(EAX=14H, ECX=0):EBX[bit 1] = 1 (Cycle-Accurate Mode is 
Supported), otherwise the CycThresh field is reserved to 0. See Section 
31.2.7. 


Each bit position in this field represents 1 encoding value in the 4-bit 
CycThresh field (ie, bit O is associated with encoding value 0). For each 
bit: 

1: CycThresh can be assigned the associated encoding value. 

0: CycThresh cannot be assigned to the associated encoding value. A 
write to CycThresh with unsupported encoding will cause #GP fault. 


31:16 


Bitmap of supported 
Configurable PSB Frequency 
encoding 


The non-zero bits indicate the map of supported encoding values for 
the IA32_RTIT_CTLPSBFreq field. This applies only if 
CPUID.(EAX=14H, ECX=0):EBX[bit 1] = 1 (Configurable PSB is 
supported), otherwise the PSBFreq field is reserved to 0. See Section 
31.2.7. 


Each bit position in this field represents 1 encoding value in the 4-bit 
PSBFreq field (ie, bit 0 is associated with encoding value 0). For each 
bit: 

1: PSBFreg can be assigned the associated encoding value. 


0: PSBFreq cannot be assigned to the associated encoding value. A 
write to PSBFreq with unsupported encoding will cause #GP fault. 


ECX 


31:0 


Reserved 


EDX 


31:0 


Reserved 


31-28 Vol. 3C 


INTEL® PROCESSOR TRACE 


31.3.1.1 Packet Decoding of RIP versus LIP 


FUP, TIP, TIP.PGE, and TIP.PGE packets can contain an instruction pointer (IP) payload. On some processor gener- 
ations, this payload will be an effective address (RIP), while on others this will be a linear address (LIP). In the 
former case, the payload is the offset from the current CS base address, while in the latter it is the sum of the offset 
and the CS base address (Note that in real mode, the CS base address is the value of CS< <4, while in protected 
mode the CS base address is the base linear address of the segment indicated by the CS register.). Which IP type 
is in use is indicated by enumeration (see CPUID.(EAX=14H, ECX=0):€CX.LIP[bit 31] in Table 31-11). 


For software that executes while the CS base address is 0 (including all software executing in 64-bit mode), the 
difference is indistinguishable. A trace decoder must account for cases where the CS base address is not 0 and the 
resolved LIP will not be evident in a trace generated on a CPU that enumerates use of RIP. This is likely to cause 
problems when attempting to link the trace with the associated binaries. 


Note that IP comparison logic, for IP filtering and TraceStop range calculation, is based on the same IP type as 
these IP packets. For processors that output RIP, the IP comparison mechanism is also based on RIP, and hence on 
those processors RIP values should be written to IA32_RTIT_ADDRn_[AB] MSRs. This can produce differing 
behavior if the same trace configuration setting is run on processors reporting different IP types, i.e. 
CPUID.(EAX=1 4H, ECX=0):ECX.LIP[bit 31]. Care should be taken to check CPUID when configuring IP filters. 


31.3.1.2 Model Specific Capability Restrictions 


Some processor generations impose restrictions that prevent use of LBRs/BTS/BTM/LERs when software has 
enabled tracing with Intel Processor Trace. On these processors, when TraceEn is set, updates of LBR, BTS, BTM, 
LERs are suspended but the states of the corresponding IA32_DEBUGCTL control fields remained unchanged as if 
it were still enabled. When TraceEn is cleared, the LBR array is reset, and LBR/BTS/BTM/LERs updates will resume. 
Further, reads of these registers will return 0, and writes will be dropped. 


The list of MSRs whose updates/accesses are restricted follows. 

¢ MSR_LASTBRANCH_x_TO_IP, MSR_LASTBRANCH_x_FROM_IP, MSR_LBR_INFO_x, MSR_LASTBRANCH_TOS 
e¢ MSR_LER_FROM_LIP, MSR_LER_TO_LIP 

¢ MSR_LBR_SELECT 


For processor with CPUID DisplayFamily_DisplayModel signature of 06_3DH, 06_47H, 06_4EH, 06_4FH, 06_56H 
and 06_5EH, the use of Intel PT and LBRs are mutually exclusive. 


31.3.2 | Enabling and Configuration of Trace Packet Generation 


To configure trace packets, enable packet generation, and capture packets, software starts with using CPUID 
instruction to detect its feature flag, CPUID.(EAX=07H,ECX=0H):EBX[bit 25] = 1; followed by enumerating the 
capabilities described in Section 31.3.1. 


Based on the capability queried from Section 31.3.1, software must configure a number of model-specific regis- 
ters. This section describes programming considerations related to those MSRs. 


31.3.2.1 Enabling Packet Generation 


When configuring and enabling packet generation, the IA32_RTIT_CTL MSR should be written after any other Intel 
PT MSRs have been written, since writes to the other configuration MSRs cause a general-protection fault (#GP) if 
TraceEn = 1. If a prior trace collection context is not being restored, then software should first clear 
IA32_RTIT_STATUS. This is important since the Stopped, and Error fields are writable; clearing the MSR clears any 
values that may have persisted from prior trace packet collection contexts. See Section 31.2.7.2 for details of 
packets generated by setting TraceEn to 1. 


If setting TraceEn to 1 causes an operational error (see Section 31.3.9), there may be a delay after the WRMSR 
completes before the error is signaled in the IA32_RTIT_STATUS MSR. 


While packet generation is enabled, the values of some configuration MSRs (e.g., IA32_RTIT_STATUS and 
IA32_RTIT_OUTPUT_*) are transient, and reads may return values that are out of date. Only after packet genera- 
tion is disabled (by clearing TraceEn) do reads of these MSRs return reliable values. 
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31.3.2.2 Disabling Packet Generation 


After disabling packet generation by clearing IA32_RTIT_CTL, it is advisable to read the IA32_RTIT_STATUS MSR 
(Section 31.2.7.4): 


e If the Error bit is set, an operational error was encountered, and the trace is most likely compromised. Software 
should check the source of the error (by examining the output MSR values), correct the source of the problem, 
and then attempt to gather the trace again. For details on operational errors, see Section 31.3.9. Software 
should clear IA32_RTIT_STATUS.Error before re-enabling packet generation. 


¢ If the Stopped bit is set, software execution encountered an IP TraceStop (see Section 31.2.4.3) or the ToPA 
Stop condition (see “ToPA STOP” in Section 31.2.6.2) before packet generation was disabled. 


31.3.3. Flushing Trace Output 


Packets are first buffered internally and then written out asynchronously. To collect packet output for post- 
processing, a collector needs first to ensure that all packet data has been flushed from internal buffers. Software 
can ensure this by stopping packet generation by clearing IA32_RTIT_CTL.TraceEn (see “Disabling Packet Genera- 
tion” in Section 31.2.7.2). 


When software clears IA32_RTIT_CTL.TraceEn to flush out internally buffered packets, the logical processor issues 
an SFENCE operation which ensures that WC trace output stores will be ordered with respect to the next store, or 
serializing operation. A subsequent read from the same logical processor will see the flushed trace data, while a 
read from another logical processor should be preceded by a store, fence, or architecturally serializing operation on 
the tracing logical processor. 

When the flush operations complete, the IA32_RTIT_OUTPUT_* MSR values indicate where the trace ended. While 
TraceEn is set, these MSRs may hold stale values. Further, if a TOPA region with INT=1 is filled, meaning a ToPA PMI 
has been triggered, IA32_PERF_GLOBAL_STATUS.Trace_ToPA_PMI[55] will be set by the time the flush completes. 


31.3.4 Warm Reset 


The MSRs software uses to program Intel Processor Trace are cleared after a power-on RESET (or cold RESET). On 
a warm RESET, the contents of those MSRs can retain their values from before the warm RESET with the exception 
that IA32_RTIT_CTL.TraceEn will be cleared (which may have the side effect of clearing some bits in 
IA32_RTIT_STATUS). 


31.3.5 Context Switch Consideration 


To facilitate construction of instruction execution traces at the granularity of a software process or thread context, 
software can save and restore the states of the trace configuration MSRs across the process or thread context 
switch boundary. The principle is the same as saving and restoring the typical architectural processor states across 
context switches. 


31.3.5.1. Manual Trace Configuration Context Switch 


The configuration can be saved and restored through a sequence of instructions of RDMSR, management of MSR 
content and WRMSR. To stop tracing and to ensure that all configuration MSRs contain stable values, software must 
clear IA32_RTIT_CTL.TraceEn before reading any other trace configuration MSRs. The recommended method for 
saving trace configuration context manually follows: 


1. RDMSR IA32_RTIT_CTL, save value to memory 
2. WRMSR IA32_RTIT_CTL with saved value from RDMSR above and TraceEn cleared 


3. RDMSR all other configuration MSRs whose values had changed from previous saved value, save changed 
values to memory 
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When restoring the trace configuration context, IA32_RTIT_CTL should be restored last: 


1. Read saved configuration MSR values, aside from IA32_RTIT_CTL, from memory, and restore them with 
WRMSR 


2. Read saved JA32_RTIT_CTL value from memory, and restore with WRMSR. 


31.3.5.2 Trace Configuration Context Switch Using XSAVES/XRSTORS 


On processors whose XSAVE feature set supports XSAVES and XRSTORS, the Trace configuration state can be 
saved using XSAVES and restored by XRSTORS, in conjunction with the bit field associated with supervisory state 
component in IA32_XSS. See Chapter 13, “Managing State Using the XSAVE Feature Set” of Inte/® 64 and IA-32 
Architectures Software Developer’s Manual, Volume 1. 


The layout of the trace configuration component state in the XSAVE area is shown in Table 31-13.! 


Table 31-13. Memory Layout of the Trace Configuration State Component 


Offset within Field Offset within Field 
Component Area Component Area 
OH IA32_RTIT_CTL 08H IA32_RTIT_OUTPUT_BASE 
10H IA32_RTIT_OUTPUT_MASK_PTRS 18H IA32_RTIT_STATUS 
20H IA32_RTIT_CR3_MATCH 28H IA32_RTIT_ADDRO_A 
30H 1A32_RTIT_ADDRO_B 38H 1A32_RTIT_ADDR1_A 
40H IA32_RTIT_ADDR1_B 48H-End Reserved 


The IA32_XSS MSR is zero coming out of RESET. Once IA32_XSS[bit 8] is set, system software operating at CPL= 
0 can use XSAVES/XRSTORS with the appropriate requested-feature bitmap (RFBM) to manage supervisor state 
components in the XSAVE map. See Chapter 13, “Managing State Using the XSAVE Feature Set” of Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 1. 


31.3.6 Cycle-Accurate Mode 


Intel PT can be run in a cycle-accurate mode which enables CYC packets (see Section 31.4.2.14) that provide low- 
level information in the processor core clock domain. This cycle counter data in CYC packets can be used to 
compute IPC (Instructions Per Cycle), or to track wall-clock time on a fine-grain level. 


To enable cycle-accurate mode packet generation, software should set IA32_RTIT_CTL.CYCEn=1. It is recom- 
mended that software also set TSCEn=1 anytime cycle-accurate mode is in use. With this, all CYC-eligible packets 
will be preceded by a CYC packet, the payload of which indicates the number of core clock cycles since the last CYC 
packet. In cases where multiple CYC-eligible packets are generated in a single cycle, only a single CYC will be 
generated before the CYC-eligible packets, otherwise each CYC-eligible packet will be preceded by its own CYC. The 
CYC-eligible packets are: 


* TNT, TIP, TIP.PGE, TIP.PGD, MODE.EXEC, MODE.TSX, PIP, VMCS, OVF, MTC, TSC, PTWRITE, EXSTOP 


TSC packets are generated when there is insufficient information to reconstruct wall-clock time, due to tracing 
being disabled (TriggerEn=0), or power down scenarios like a transition to a deep-sleep MWAIT C-state. In this 
case, the CYC that is generated along with the TSC will indicate the number of cycles actively tracing (those 
powered up, with TriggerEn=1) executed between the last CYC packet and the TSC packet. And hence the amount 
of time spent while tracing is inactive can be inferred from the difference in time between that expected based on 
the CYC value, and the actual time indicated by the TSC. 


Additional CYC packets may be sent stand-alone, so that the processor can ensure that the decoder is aware of the 
number of cycles that have passed before the internal hardware counter wraps, or is reset due to other micro- 
architectural condition. There is no guarantee at what intervals these standalone CYC packets will be sent, except 
that they will be sent before the wrap occurs. An illustration is given below. 


1. Table 31-13 documents support for the MSRs defining address ranges 0 and 1. Processors that provide XSAVE support for Intel Processor 
Trace support only those address ranges. 
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Example 31-1. An Illustrative CYC Packet Example 


Time (cycles) | Instruction Snapshot Generated Packets Comment 

x call Y%eax CYC(?), TIP ?€lapsed cycles from the previous CYC unknown 

x+2 call %ebx CYC(2), TIP 1 byte CYC packet; 2 cycles elapsed from the previous CYC 
x+8 jnz Foo (not taken) CYC(6) 1 byte CYC packet 

x+9 ret (compressed) 

x+12 jnz Bar (taken) 

x+16 ret (uncompressed) TNT, CYC(8), TIP 1 byte CYC packet 

x+4111 CYC(4095) 2 byte CYC packet 

x + 12305 CYC(8194) 3 byte CYC packet 

xX + 16332 mov cr3, %ebx CYC(4027), PIP 2 byte CYC packet 


31.3.6.1. Cycle Counter 


The cycle counter is implemented in hardware (independent of the time stamp counter or performance monitoring 
counters), and is a simple incrementing counter that does not saturate, but rather wraps. The size of the counter is 
implementation specific. 


The cycle counter is reset to zero any time that TriggerEn is cleared, and when a CYC packet is sent. The cycle 
counter will continue to count when ContextEn or FilterEn are cleared, and cycle packets will still be generated. It 
will not count during sleep states that result in Intel PT logic being powered-down, but will count up to the point 
where clocks are disabled, and resume counting once they are re-enabled. 


31.3.6.2 Cycle Packet Semantics 
Cycle-accurate mode adheres to the following protocol: 
¢ All packets that precede a CYC packet represent instructions or events that took place before the CYC time. 


¢ All packets that follow a CYC packet represent instructions or events that took place at the same time as, or 
after, the CYC time. 


¢ The CYC-eligible packet that immediately follows a CYC packet represents an instruction or event that took 
place at the same time as the CYC time. 


These items above give the decoder a means to apply CYC packets to a specific instruction in the assembly stream. 
Most packets represent a single instruction or event, and hence the CYC packet that precedes each of those packets 
represents the retirement time of that instruction or event. In the case of TNT packets, up to 6 conditional branches 
and/or compressed RETs may be contained in the packet. In this case, the preceding CYC packet provides the 
retirement time of the first branch in the packet. It is possible that multiple branches retired in the same cycle as 
that first branch in the TNT, but the protocol will not make that obvious. Also note that a MTC packet could be 
generated in the same cycle as the first JCC in the TNT packet. In this case, the CYC would precede both the MTC 
and the TNT, and apply to both. 


Note that there are times when the cycle counter will stop counting, though cycle-accurate mode is enabled. After 
any such scenario, a CYC packet followed by TSC packet will be sent. See Section 31.8.3.2 to understand how to 
interpret the payload values 


Multi-packet Instructions or Events 


Some operations, such as interrupts or task switches, generate multiple packets. In these cases, multiple CYC 
packets may be sent for the operation, preceding each CYC-eligible packet in the operation. An example, using a 
task switch on a software interrupt, is shown below. 
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Example 31-2. An Example of CYC in the Presence of Multi-Packet Operations 


Time (cycles) Instruction Snapshot Generated Packets 
X jnz Foo (not taken) CYC(?), 
Xx+2 ret (compressed) 
x+8 jnz Bar (taken) 
x+9 jmp %eax TNT, CYC(9), TIP 
x+12 jnz Bar (not taken) CYC(3) 
x + 32 int3 (task gate) TNT, FUP, CYC(10), PIP, CYC(20), MODE.Exec, TIP 


31.3.6.3 Cycle Thresholds 


Software can opt to reduce the frequency of cycle packets, a trade-off to save bandwidth and intrusion at the 
expense of precision. This is done by utilizing a cycle threshold (see Section 31.2.7.2). 


IA32_RTIT_CTL.CycThresh indicates to the processor the minimum number of cycles that must pass before the 
next CYC packet should be sent. If this value is 0, no threshold is used, and CYC packets can be sent every cycle in 
which a CYC-eligible packet is generated. If this value is greater than 0, the hardware will wait until the associated 
number of cycles have passed since the last CYC packet before sending another. CPUID provides the threshold 
options for CycThresh, see Section 31.3.1. 


Note that the cycle threshold does not dictate how frequently a CYC packet will be posted, it merely assigns the 
maximum frequency. If the cycle threshold is 16, a CYC packet can be posted no more frequently than every 16 
cycles. However, once that threshold of 16 cycles has passed, it still requires a new CYC-eligible packet to be gener- 
ated before a CYC will be inserted. Table 31-14 illustrates the threshold behavior. 


Table 31-14. An Illustrative CYC Packet Example 


Time (cycles) Instruction Snapshot Uuoli 
0 16 32 64 
x jmp %eax CYC, TIP CYC, TIP CYC, TIP CYC, TIP 
x+9 call %ebx CYC, TIP TIP TIP TIP 
x+15 call %ecx CYC, TIP TIP TIP TIP 
x + 30 jmp %edx CYC, TIP CYC, TIP TIP TIP 
xX + 38 mov cr3, %eax CYC, PIP PIP CYC, PIP PIP 
x + 46 jmp [%eax] CYC, TIP CYC, TIP TIP TIP 
x + 64 call %edx CYC, TIP CYC, TIP TIP CYC,TIP 
x+7] jmp %edx CYC, TIP TIP CYC,TIP TIP 


31.3.7. Decoder Synchronization (PSB+) 


The PSB packet (Section 31.4.2.17) serves as a synchronization point for a trace-packet decoder. It is a pattern in 
the trace log for which the decoder can quickly scan to align packet boundaries. No legal packet combination can 
result in such a byte sequence. As such, it serves as the starting point for packet decode. To decode a trace log 
properly, the decoder needs more than simply to be aligned: it needs to know some state and potentially some 
timing information as well. The decoder should never need to retain any information (e.g., LastIP, call stack, 
compound packet event) across a PSB; all compound packet events will be completed before a PSB, and any 
compression state will be reset. 

When a PSB packet is generated, it is followed by a PSBEND packet (Section 31.4.2.18). One or more packets may 
be generated in between those two packets, and these inform the decoder of the current state of the processor. 
These packets, known collectively as PSB+, should be interpreted as “status only”, since they do not imply any 
change of state at the time of the PSB, nor are they associated directly with any instruction or event. Thus, the 
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normal binding and ordering rules that apply to these packets outside of PSB+ can be ignored when these packets 
are between a PSB and PSBEND. They inform the decoder of the state of the processor at the time of the PSB. 


PSB+ can include: 
¢ Timestamp (TSC), if IA32_RTIT_CTL.TSCEn=1. 
° Timestamp-MTC Align (TMA), if IA32_RTIT_CTL.TSCEn=1 && IA32_RTIT_CTL.MTCEn=1. 


° Paging Information Packet (PIP), if ContextEn=1 and IA32_RTIT_CTL.OS=1. The non-root bit (NR) is set if the 
logical processor is in VMX non-root operation and the “conceal VMX from PT” VM-execution control is 0. 


° VMCS packet, if either the logical is in VMX root operation or the logical processor is in VMX non-root operation 
and the “conceal VMX from PT” VM-execution control is 0. 


* Core Bus Ratio (CBR). 

° MODE.TSX, if ContextEn=1 and BranchEn = 1. 
° MODE.Exec, if PacketEn=1. 

¢ Flow Update Packet (FUP), if PacketEn=1. 


PSB is generated only when TriggerEn=1; hence PSB+ has the same dependencies. The ordering of packets within 
PSB+ is not fixed. Timing packets such as CYC and MTC may be generated between PSB and PSBEND, and their 
meanings are the same as outside PSB+. 


A PSB+ can be lost in some scenarios. If IA32_RTIT_STATUS. TriggerEn is cleared just as the PSB threshold is 
reached, e.g., due to TraceEn being cleared, the PSB+ may not be generated. On processors that support PSB pres- 
ervation (CPUID.(EAX=14H, ECX=0):EBX[bit 6] = 1), setting IA32_RTIT_CTL.InjectPsbPmiOnEnable[56] = 1 will 
ensure that a PSB+ that is pending at the time PT is disabled will be recorded by setting 
IA32_RTIT_STATUS.PendPSB[6] = 1. A PSB will be inserted, and PendPSB cleared, when PT is later re-enabled 
while PendPSB = 1. 


Note that an overflow can occur during PSB+, and this could cause the PSBEND packet to be lost. For this reason, 
the OVF packet should also be viewed as terminating PSB+. If IA32_RTIT_STATUS.TriggerEn is cleared just as the 
PSB threshold is reached, the PSB+ may not be generated. TriggerEn can be cleared by a WRMSR that clears 
IA32_RTIT_CTL.TraceEn, a VM-exit that clears IA32_RTIT_CTL.TraceEn, an #SMI, or any time that either 
IA32_RTIT_STATUS. Stopped is set (e.g., by a TraceStop or ToPA stop condition) or IA32_RTIT_STATUS.Error is set 
(e.g., by an Intel PT output error). On processors that support PSB preservation (CPUID.(EAX=14H, 
ECX=0):EBX[bit 6] = 1), setting IA32_RTIT_CTL.InjectPsbPmiOnEnable[56] = 1 will ensure that a PSB+ that is 
pending at the time PT is disabled will be recorded by setting IA32_RTIT_STATUS.PendPSB[6] = 1. A PSB will then 
be pended when the saved PT context is later restored. 


31.3.8 Internal Buffer Overflow 


In the rare circumstances when new packets need to be generated but the processor’s dedicated internal buffers 
are all full, an “internal buffer overflow” occurs. On such an overflow packet generation ceases (as packets would 
need to enter the processor's internal buffer) until the overflow resolves. Once resolved, packet generation 
resumes. 


When the buffer overflow is cleared, an OVF packet (Section 31.4.2.16) is generated, and the processor ensures 
that packets which follow the OVF are not compressed (IP compression or RET compression) against packets that 
were lost. 


If IA32_RTIT_CTL.BranchEn = 1, the OVF packet will be followed by a FUP if the overflow resolves while Pack- 
etEn=1. If the overflow resolves while PacketEn = 0 no packet is generated, but a TIP.PGE will naturally be gener- 
ated later, once PacketEn = 1. The payload of the FUP or TIP.PGE will be the Current IP of the first instruction upon 
which tracing resumes after the overflow is cleared. If the overflow resolves while PacketEn=1, only timing packets 
may come between the OVF and the FUP. If the overflow resolves while PacketEn=0, any other packets that are not 
dependent on PacketEn may come between the OVF and the TIP.PGE. 


31.3.8.1 Overflow Impact on Enables 


The address comparisons to ADDRn ranges, for IP filtering and TraceStop (Section 31.2.4.3), continue during a 
buffer overflow, and TriggerEn, ContextEn, and FilterEn may change during a buffer overflow. Like other packets, 
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however, any TIP.PGE or TIP.PGD packets that would have been generated will be lost. Further, 
IA32_RTIT_STATUS.PacketByteCnt will not increment, since it is only incremented when packets are generated. 


If a TraceStop event occurs during the buffer overflow, IA32_RTIT_STATUS.Stopped will still be set, tracing will 
cease as a result. However, the TraceStop packet, and any TIP.PGD that result from the TraceStop, may be 
dropped. 


31.3.8.2 Overflow Impact on Timing Packets 


Any timing packets that are generated during a buffer overflow will be dropped. If only a few MTC packets are 
dropped, a decoder should be able to detect this by noticing that the time value in the first MTC packet after the 
buffer overflow incremented by more than one. If the buffer overflow lasted long enough that 256 MTC packets are 
lost (and thus the MTC packet ‘wraps’ its 8-bit CTC value), then the decoder may be unable to properly understand 
the trace. This is not an expected scenario. No CYC packets are generated during overflow, even if the cycle counter 
wraps. 


Note that, if cycle-accurate mode is enabled, the OVF packet will generate a CYC packet. Because the cycle counter 
counts during overflows, this CYC packet can provide the duration of the overflow. However, there is a risk that the 
cycle counter wrapped during the overflow, which could render this CYC misleading. 


31.3.9 Operational Errors 


Errors are detected as a result of packet output configuration problems, which can include output alignment issues, 
ToPA reserved bit violations, or overlapping packet output with restricted memory. See “ToPA Errors” in Section 
31.2.6.2 for details on ToPA errors, and Section 31.2.6.4 for details on restricted memory errors. Operational 
errors are only detected and signaled when TraceEn=1. 


When an operational error is detected, tracing is disabled and the error is logged. Specifically, 
IA32_RTIT_STATUS.Error is set, which will cause IA32_RTIT_STATUS.TriggerEn to be 0. This will disable genera- 
tion of all packets. Some causes of operational errors may lead to packet bytes being dropped. 


It should be noted that the timing of error detection may not be predictable. Errors are signaled when the 
processor encounters the problematic configuration. This could be as soon as packet generation is enabled but 
could also be later when the problematic entry or field needs to be used. 


Once an error is signaled, software should disable packet generation by clearing TraceEn, diagnose and fix the error 
condition, and clear IA32_RTIT_STATUS.Error. At this point, packet generation can be re-enabled. 


31.4 TRACE PACKETS AND DATA TYPES 


This section details the data packets generated by Intel Processor Trace. It is useful for developers writing the 
interpretation code that will decode the data packets and apply it to the traced source code. 


31.4.1 Packet Relationships and Ordering 


This section introduces the concept of packet “binding”, which involves determining the IP in a binary disassembly 
at which the change indicated by a given packet applies. Some packets have the associated IP as the payload (FUP, 
TIP), while for others the decoder need only search for the next instance of a particular instruction (or instructions) 
to bind the packet (TNT). However, in many cases, the decoder will need to consider the relationship between 
packets, and to use this packet context to determine how to bind the packet. 


Section 31.4.1.1 below provides detailed descriptions of the packets, including how packets bind to IPs in the 
disassembly, to other packets, or to nothing at all. Many packets listed are simple to bind, because they are gener- 
ated in only a few scenarios. Those that require more consideration are typically part of “compound packet events”, 
such as interrupts, exceptions, and some instructions, where multiple packets are generated by a single operation 
(instruction or event). These compound packet events frequently begin with a FUP to indicate the source address 
(if it is not clear from the disassembly), and are concluded by a TIP or TIP.PGD packet that indicates the destination 
address (if one is provided). In this scenario, the FUP is said to be “coupled” with the TIP packet. 
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Other packets could be in between the coupled FUP and TIP packet. Timing packets, such as TSC, MTC, CYC, or 
CBR, could arrive at any time, and hence could intercede in a compound packet event. If an operation changes CR3 
or the processor’s mode of execution, a state update packet (i.e., PIP or MODE) is generated. The state changes 
indicated by these intermediate packets should be applied at the IP of the TIP* packet. A summary of compound 
packet events is provided in Table 31-15; see Section 31.4.1.1 for more per-packet details and Section 31.7 for 
more detailed packet generation examples. 


Table 31-15. Compound Packet Event Summary 


Event Type Beginning Middle End Comment 
Unconditional, | FUP or none Any combination | TIP or TIRPGD | FUP only for asynchronous events. Order of middle packets 
uncompressed of PIP, VMCS, may vary. 
control-flow MODE.Exec, or PIP/VMCS/MODE only if the operation modifies the state 
transfer none tracked by these respective packets. 
TSX Update MODE.TSX, and | None TIP, TIP.PGD, or | FUP 
(FUP or none) none TIP/TIP-PGD only for TSX abort cases. 
Overflow OVF PSB, PSBEND, or | FUP or TIPPGE | FUP if overflow resolves while ContextEn=1, else TIP.PGE. 
none 


31.4.1.1 Packet Blocks 


Packet blocks are a means to dump one or more groups of state values. Packet blocks begin with a Block Begin 
Packet (BBP), which indicates what type of state is held within the block. Following each BBP there may be one or 
more Block Item Packets (BIPs), which contain the state values. The block is terminated by either a Block End 
Packet (BEP) or another BBP indicating the start of a new block. 


The BIP packet includes an ID value that, when combined with the Type field from the BBP that preceded it, 
uniquely identifies the state value held in the BIP payload. The size of each BIP packet payload is provided by the 
Size field in the preceding BBP packet. 


Each block type can have up to 32 items defined for it. There is no guarantee, however, that each block of that type 
will hold all 32 items. For more details on which items to expect, see documentation on the specific block type of 
interest. 


See the BBP packet description (Section 31.4.2.26) for details on packet block generation scenarios. 


Packet blocks are entirely generated within an instruction or between instructions, which dictates the types of 
packets (aside from BIPs) that may be seen within a packet block. Packets that indicate control flow changes, or 
other indication of instruction completion, cannot be generated within a block. These are listed in the following 
table. Other packets, including timing packets, may occur between BBP and BEP. 


Table 31-16. Packets Forbidden Between BBP and BEP 


TNT 


TIP, TIP.PGE, TIP.PGD 


MODE.Exec, MODE.TSX 


PIP, VMCS 


TraceStop 


PSB, PSBEND 


PTW 


MWAIT 


It is possible to encounter an internal buffer overflow in the middle of a block. In such a case, it is guaranteed that 
packet generation will not resume in the middle of a block, and hence the OVF packet terminates the current block. 
Depending on the duration of the overflow, subsequent blocks may also be lost. 
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When a Block Begin Packet (BBP) is encountered, the decoder will need to decode some packets within the block 
differently from those outside a block. The Block Item Packet (BIP) header byte has the same encoding as a TNT 
packet outside of a block, but must be treated as a BIP header (with following payload) within one. 


When an OVF packet is encountered, the decoder should treat that as a block ending condition. Packet generation 
will not resume within a block. 


31.4.2 Packet Definitions 


The following description of packet definitions are in tabular format. Figure 31-3 explains how to interpret them. 
Packet bits listed as “RSVD” are not guaranteed to be 0. 


i 


Byte Number Bit Number Crean Payload in White 


Header bits 


Name 


Packet name } { 


Packet Format 


0 
1 
2 


Description of fields 


o| -| Oo] N 
=| =| =| 
o| oO] O] u 
=| =] =] ~ 
= -Sroi= 
= 


Oo} OC; CO; W 


oO; oO; —| & 


Dependencies | Depends on packet generation con- | Generation Scenario | Which instructions, events, or other 
figuration enable controls or other scenarios can cause this packet to be 
bits (Section 31.2.5). generated. 

Description Description of the packet, including the purpose it serves, meaning of the information or payload, etc 

Application How a decoder should apply this packet. It may bind to a specific instruction from the binary, or to 


another packet in the stream, or have other implications on decode 


Figure 31-3. Interpreting Tabular Definition of Packet Format 
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31.4.2.1  Taken/Not-taken (TNT) Packet 


Table 31-17. TNT Packet Definition 


Name 


Taken/Not-taken (TNT) Packet 


Packet Format 


v 6 


5 4 


3 2 1 0 


0 1 By 


Bo B3 


Ba Bs Be 0 Short TNT 


B1...BN represent the last N conditional branch or compressed RET (Section 31.4.2.2) results, such that B1 is oldest 
and BN is youngest. The short TNT packet can contain from 1 to 6 TNT bits. The long TNT packet can contain from 


1 to 47 TNT bits. 


i 6 5 4 2) 2 1 0 
0 0) 0 0 0 0 1 0 Long TNT 
1 1 0 1 0 0 0 1 1 
2 Bao Bay Baz Ba3 Bag Bas Bag Ba7 
: B32 B33 B34 B3s B3g B37 B3g B39 
Hs Bo4 Bas B26 B27 Bag B29 B3o B3) 
5 Big Bi7 Big Big Bzo Bay B22 B23 
6 Bg Bg Bio Bry Bi2 Bi3 Big Bis 
7 1 By Bo B3 By Bs Bg Bz 


Irrespective of how many TNT bits is in a packet, the last valid TNT bit is followed by a trailing 1, or Stop bit, as 
shown above. If the TNT packet is not full (fewer than 6 TNT bits for the Short TNT, or fewer than 47 TNT bits for 
the Long TNT), the Stop bit moves up, and the trailing bits of the packet are filled with Os. Examples of these 
“partial TNTs” are shown below. An implementation may choose to use long TNTs, short TNTs, or both. 


7 6 5 4 3 2 1 0 

0 0 0 1 By Bo B3 By 0 Short TNT 
7 6 5 4 5] 2 1 0 

0 0 0 0 0 0 1 0 Long TNT 

1 1 0 1 0 0 0 1 1 

2 Ba Bas Bag B27 Bog Bag B30 B31 

3 Big Bi7 Big Big B20 Bay B22 B23 

a Bg Bg Bio Bi Bi2 Bi3 Bia Bis 

5 1 B, Bo B3 By Bs Be BZ 

6 0 0 0 0 0 0 0 0 

7 0 0 0 0 0 0 0 0 

Dependencies | PacketEn Generation On a conditional branch or compressed RET, if it fills the TNT. 
Scenario Also, partial TNTs may be generated at any time, as a result of 


other packets being generated, or certain micro-architectural 
conditions occurring, before the TNT is full. 
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Table 31-17. TNT Packet Definition (Contd.) 


Description 


Provides the taken/not-taken results for the last 1..6 (Short TNT) or 1..47 (Long TNT) conditional branches (jcc, 
J*CXZ, or LOOP) or compressed RETs (Section 31.4.2.2). The TNT payload bits should be interpreted as follows: 
«1 indicates a taken conditional branch, or a compressed RET 

* OQ indicates a not-taken conditional branch 

TNT payload bits are stored internal to the processor in a TNT buffer, until either the buffer is filled or another 
packet is to be generated. In either case a TNT packet holding the buffered bits will be emitted, and the TNT buffer 
will be marked as empty. 


Application 


Each valid payload bit (that is, bits between the header bits and the trailing Stop bit) applies to an upcoming condi- 
tional branch or RET instruction. Once a decoder consumes a TNT packet with N valid payload bits, these bits should 
be applied to (and hence provide the destination for) the next N conditional branches or RETs. 


31.4.2.2 Target IP (TIP) Packet 


Table 31-18. IP Packet Definition 


Name 


Target IP (TIP) Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0) IPBytes 0) 1 1 0 
1 TargetIP[7:0] 
2 TargetIP[15:8] 
3 TargetIP[23:16] 
4 TargetIP[3 1:24] 
5 TargetIP[39:32] 
6 TargetIP[47:40] 
7 TargetIP[55:48] 
8 TargetIP[63:56] 
Dependencies | PacketEn Generation Sce- Indirect branch (including un-compressed RET), far branch, interrupt, 
nario exception, INIT, SIPI, VM exit, VM entry, TSX abort, EENTER, EEXIT, ERE- 
SUME, AEX". 
Description Provides the target for some control flow transfers 
Application Anytime a TIP is encountered, it indicates that control was transferred to the IP provided in the payload. 
The source of this control flow change, and hence the IP or instruction to which it binds, depends on the packets 
that precede the TIP. If a TIP is encountered and all preceding packets have already been bound, then the TIP will 
apply to the upcoming indirect branch, far branch, or VMRESUME. However, if there was a preceding FUP that 
remains unbound, it will bind to the TIP. Here, the TIP provides the target of an asynchronous event or TSX abort 
that occurred at the IP given in the FUP payload. Note that there may be other packets, in addition to the FUP, which 
will bind to the TIP packet. See the packet application descriptions for other packets for details. 
NOTES: 


1. EENTER, EEXIT, ERESUME, AEX would be possible only for a debug enclave. 


IP Compression 


The IP payload in a TIP. FUP, TIP.PGE, or TIP.PGD packet can vary in size, based on the mode of execution, and the 
use of IP compression. IP compression is an optional compression technique the processor may choose to employ 
to reduce bandwidth. With IP compression, the IP to be represented in the payload is compared with the last IP 

sent out, via any of FUP, TIP, TIP.PGE, or TIP.PGD. If that previous IP had the same upper (most significant) address 
bytes, those matching bytes may be suppressed in the current packet. The processor maintains an internal state 
of the “Last IP” that was encoded in trace packets, thus the decoder will need to keep track of the “Last IP” state in 
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software, to match fidelity with packets generated by hardware. “Last IP” is initialized to zero, hence if the first IP 
in the trace may be compressed if the upper bytes are zeroes. 


The “IPBytes” field of the IP packets (FUP, TIP, TIP.PGE, TIP.PGD) serves to indicate how many bytes of payload are 
provided, and how the decoder should fill in any suppressed bytes. The algorithm for reconstructing the IP for a 
TIP/FUP packet is shown in the table below. 


Table 31-19. FUP/TIP IP Reconstruction 


IPBytes Uncompressed IP Value 
63:56 55:48 47:40 39:32 31:24 23:16 15:8 7:0 
000b None, IP is out of context 
001b Last IP[63:16] IP Payload[15:0] 
010b Last IP[63:32] IP Payload[31:0] 
011b IP Payload[47] extended IP Payload[47:0] 
100b Last IP [63:48] IP Payload[47:0] 
101b Reserved 
110b IP Payload[63:0] 
111b Reserved 


The processor-internal Last IP state is guaranteed to be reset to zero when a PSB is sent out. This means that the 
IP that follows the PSB with either be un-compressed (011b or 110b, see Table 31-19), or compressed against 
zero. 


At times, “IPbytes” will have a value of 0. As shown above, this does not mean that the IP payload matches the full 
address of the last IP, but rather that the IP for this packet was suppressed. This is used for cases where the IP that 
applies to the packet is out of context. An example is the TIP.PGD sent on a SYSCALL, when tracing only USR code. 
In that case, no TargetIP will be included in the packet, since that would expose an instruction point at CPL = 0. 
When the IP payload is suppressed in this manner, Last IP is not cleared, and instead refers to the last IP packet 
with a non-zero IPBytes field. 


On processors that support a maximum linear address size of 32 bits, IP payloads may never exceed 32 bits 
(IPBytes <= 010b). 


Indirect Transfer Compression for Returns (RET) 


In addition to IP compression, TIP packets for near return (RET) instructions can also be compressed. If the RET 
target matches the next IP of the corresponding CALL, then the TIP packet is unneeded, since the decoder can 
deduce the target IP by maintaining a CALL/RET stack of its own. 


When a RET is compressed, a Taken indication is added to the TNT buffer. Because the RET generates no TIP 
packet, it also does not update the internal Last IP value, and thus the decoder should treat it the same way. If the 
RET is not compressed, it will generate a TIP packet (just like when RET compression is disabled, via 
IA32_RTIT_CTL.DisRETC). 


A CALL/RET stack can be maintained by the decoder by doing the following: 
1. Allocate space to store 64 RET targets. 


2. Fornear CALLs, push the Next IP onto the stack. Once the stack is full, new CALLs will force the oldest entry off 
the end of the stack, such that only the youngest 64 entries are stored. Note that this excludes zero-length 
CALLs, which are direct near CALLs with displacement zero (to the next IP). These CALLs typically don’t have 
matching RETs. 


3. For near RETs, pop the top (youngest) entry off the stack. This will be the expected target of the RET. 


In cases where a RET is compressed, the RET target is guaranteed to match the expected target from 3) above. If 
the target is not compressed, a TIP packet will be generated with the RET target, which may differ from the 
expected target in some cases. 
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The hardware ensures that packets read by the decoder will always have seen the CALL that corresponds to any 
compressed RET. The processor will never compress a RET across a PSB, a buffer overflow, or scenario where Pack- 
etEn=0. This means that a RET whose corresponding CALL executed while PacketEn=0, or before the last PSB, etc., 
will not be compressed. 


If the CALL/RET stack is manipulated or corrupted by software, and thereby causes a RET to transfer control to a 
target that is inconsistent with the CALL/RET stack, then the RET will not be compressed, and will produce a TIP 
packet. This can happen, for example, if software executes a PUSH instruction to push a target onto the stack, and 
a later RET uses this target. 


For processors that employ deferred TIPs (Section 31.4.2.3), an uncompressed RET will not be deferred, and hence 
will force out any accumulated TNTs or TIPs. This serves to avoid ambiguity, and make clear to the decoder whether 
the near RET was compressed, and hence a bit in the in-progress TNT should be consumed, or uncompressed, in 
which case there will be no in-progress TNT and thus a TIP should be consumed. 


Note that in the unlikely case that a RET executes in a different execution mode than the associated CALL, the 
decoder will need to model the same behavior with its CALL stack. For instance, if a CALL executes in 64-bit mode, 
a 64-bit IP value will be pushed onto the software stack. If the corresponding RET executes in 32-bit mode, then 
only the lower 32 target bits will be popped off of the stack, which may mean that the RET does not go to the CALL’s 
Next IP. This is architecturally correct behavior, and this RET could be compressed, thus the decoder should match 
this behavior. 


31.4.2.3. Deferred TIPs 


The processor may opt to defer sending out the TNT when TIPs are generated. Thus, rather than sending a partial 
TNT followed by a TIP, both packets will be deferred while the TNT accumulates more Jcc/RET results. Any number 
of TIP packets may be accumulated this way, such that only once the TNT is filled, or once another packet (e.g., 
FUP) is generated, the TNT will be sent, followed by all the deferred TIP packets, and finally terminated by the other 
packet(s) that forced out the TNT and TIP packets. Generation of many other packets (see list below) will force out 
the TNT and any accumulated TIP packets. This is an optional optimization in hardware to reduce the bandwidth 
consumption, and hence the performance impact, incurred by tracing. 


Table 31-20. TNT Examples with Deferred TIPs 
Code Flow Packets, Non-Deferred TIPS Packets, Deferred TIPS 


0x1000 cmp %rcx, 0 
0x1004 jnz Foo // not-taken 
0x1008 jmp “%rdx 

0x1308 cmp %rcx, 1 
0x130c jnz Bar // not-taken 
0x1310 cmp %rcx, 2 

0x1314 jnz Baz // taken 
0x1500 cmp %eax, 7 
0x1504 jg Exit // not-taken TNT(06010), TIP(Ox1 100) 
0x1508 jmp %r15 

0x1100 cmp %rbx, 1 
0x1104 jg Start // not-taken 
0x1108 add %rcx, %eax 


TNT(0b0), TIP(Ox1 308) 


TNT(0b0), FUP(Ox1 100), 


0x110c...// an asynchronous Interrupt arrives TIP(OxccO0) 
INThandler: TNT(0b00100), TIP(Ox1308), 
OxccOO pop %rdx TIP(Ox1 100), FUP(Ox1 10c), 


TIP(OxccO0) 
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31.4.2.4 Packet Generation Enable (TIP.PGE) Packet 


Table 31-21. TIP.PGE Packet Definition 


Name Target IP - Packet Generation Enable (TIP.PGE) Packet 
Packet Format 
7 6 5 4 3 2 1 0 

0 IPBytes 1 0 0 0 1 

1 TargetIP[7:0] 

2 TargetIP[15:8] 

3 TargetIP[23:16] 

4 TargetIP[3 1:24] 

5 TargetIP[39:32] 

6 TargetIP[47:40] 

7 TargetIP[55:48] 

8 TargetIP[63:56] 
Dependencies | PacketEn transitions to 1 Generation Any branch instruction, control flow transfer, or MOV 

Scenario CR3 that sets PacketEn, a WRMSR that enables 
packet generation and sets PacketEn 

Description Indicates that PacketEn has transitioned to 1. It provides the IP at which the tracing begins. 


This can occur due to any of the enables that comprise PacketEn transitioning from O to 1, as long as all the others 

are asserted. Examples: 

* Trigger€n: This is set on software write to set IA32_RTIT_CTL.TraceEn as long as the Stopped and Error bits in 

IA32_RTIT_STATUS are clear. The IP payload will be the Next IP of the WRMSR. 

FilterEn: This is set when software jumps into the tracing region. This region is defined by enabling IP filtering in 

IA32_RTIT_CTLADDRn_CFG, and defining the range in IA32_RTIT_ADDRn_[AB], see. Section 31.2.4.3. The 

IP payload will be the target of the branch. 

* Context€n: This is set on a CPL change, a CR3 write or any other means of changing ContextEn. The IP payload 
will be the Next IP of the instruction that changes context if it is not a branch, otherwise it will be the target of 
the branch. 


Application TIP.PGE packets bind to the instruction at the IP given in the payload. 
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31.4.2.5 Packet Generation Disable (TIP.PGD) Packet 


Table 31-22. TIP.PGD Packet Definition 


Name 


Target IP - Packet Generation Disable (TIP.PGD) Packet 


Packet Format 


Description 


7 6 5 4 3 2 1 0) 
0 IPBytes 0 0 0 0 1 
1 TargetIP[7:0] 
2 TargetIP[1 5:8] 
3 TargetIP[23:16] 
4 TargetIP[3 1:24] 
5 TargetIP[39:32] 
6 TargetIP[47:40] 
7 TargetIP[55:48] 
8 TargetIP[63:56] 
Dependencies | PacketEn transitions to | Generation Any branch instruction, control flow transfer, or MOV CR3 that clears 
0 Scenario PacketEn, a WRMSR that disables packet generation and clears PacketEn 


Indicates that PacketEn has transitioned to O. It will include the IP at which the tracing ends, unless ContextEn= 0 or 

TraceEn=0 at the conclusion of the instruction or event that cleared PacketEn. 

PacketEn can be cleared due to any of the enables that comprise PacketEn transitioning from 1 to 0. Examples: 

* TriggerEn: This is cleared on software write to clear [A32_RTIT_CTL.TraceEn, or when 

IA32_RTIT_STATUS.Stopped is set, or on operational error. The IP payload will be suppressed in this case, and the 

“IPBytes” field will have the value O. 

FilterEn: This is cleared when software jumps out of the tracing region. This region is defined by enabling IP 

filtering in |A32_RTIT_CTLADDRn_CFG, and defining the range in [A32_RTIT_ADDRn_[AB], see. Section 31.2.4.3. 

The IP payload will depend on the type of the branch. For conditional branches, the payload is suppressed 

(IPBytes = 0), and in this case the destination can be inferred from the disassembly. For any other type of branch, 

the IP payload will be the target of the branch. 

* ContextEn: This can happen on a CPL change, a CR3 write or any other means of changing ContextEn. See 
Section 31.2.4.3 for details. In this case, when ContextEn is cleared, there will be no IP payload. The “IPBytes” 
field will have value 0. 

Note that, in cases where a branch that would normally produce a TIP packet (i.e., far transfer, indirect branch, inter- 

rupt, etc) or TNT update (conditional branch or compressed RT) causes Packet€En to transition from 1 to O, the TIP or 

TNT bit will be replaced with TIP.PGD. The payload of the TIP.PGD will be the target of the branch, unless the result 

of the instruction causes TraceEn or ContextEn to be cleared (ie, SYSCALL when IA32_RTIT_CTL.OS=0, In the case 

where a conditional branch clears FilterEn and hence PacketEn, there will be no TNT bit for this branch, replaced 
instead by the TIP.PGD. 


Application 


TIP.PGD can be produced by any branch instructions, as well as some non-branch instructions, that clear PacketEn. 
When produced by a branch, it replaces any TIP or TNT update that the branch would normally produce. 

In cases where there is an unbound FUP preceding the TIP.PGD, then the TIP.PGD is part of compound operation (i.e., 
asynchronous event or TSX abort) which cleared PacketEn. For most such cases, the TIP.PGD is simply replacing a 
TIP, and should be treated the same way. The TIP.PGD may or may not have an IP payload, depending on whether 
the operation cleared ContextEn. 

If there is not an associated FUP, the binding will depend on whether there is an IP payload. If there is an IP payload, 
then the TIP.PGD should be applied to either the next direct branch whose target matches the TIP.PGD payload, or 
the next branch that would normally generate a TIP or TNT packet. If there is no IP payload, then the TIP.PGD should 
apply to the next branch or MOV CR3 instruction. 
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31.4.2.6 Flow Update (FUP) Packet 


Table 31-23. FUP Packet Definition 


Name Flow Update (FUP) Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 IPBytes 1 1 1 0 1 
il IP[7:0] 
2 IP[15:8] 
3 IP[23:16] 
4 IP[31:24] 
5 IP[39:32] 
6 IP[47:40] 
7 IP[55:48] 
8 IP[63:56] 
Dependencies | TriggerEn & ContextEn. Generation | Asynchronous Events (interrupts, exceptions, INIT, SIPI, SMI, VM exit, 
(Typically depends on Scenario #MC), PSB+, XBEGIN, XEND, XABORT, XACQUIRE, XRELEASE, EENTER, 
Branch€n and FilterEn as well, EEXIT, ERESUME, EEE, AEX,', INTO, INT1, INT3, INT n, a WRMSR that 
see Section 31.2.4, Section disables packet generation. 
31.4.2.21, and Section 
31.4.2.22 for details.) 
Description Provides the source address for asynchronous events, and some other instructions. Is never sent alone, always sent 
with an associated TIP or MODE packet, and potentially others. 
Application FUP packets provide the IP to which they bind. However, they are never standalone, but are coupled with other 
packets. 
In TSX cases, the FUP is immediately preceded by a MODE.TSX, which binds to the same IP. A TIP will follow only in 
the case of TSX aborts, see Section 31.4.2.8 for details. 
Otherwise, FUPs are part of compound packet events (see Section 31.4.1). In these compound cases, the FUP pro- 
vides the source IP for an instruction or event, while a following TIP (or TIP.PGD) packet will provide the destination 
IP. Other packets may be included in the compound event between the FUP and TIP. 
NOTES: 


1. EENTER, EEXIT, ERESUME, EEE, AEX apply only if Intel Software Guard Extensions is supported. 
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Flow Update Packet gives the source address of an instruction when it is needed. In general, branch instructions do 
not need a FUP, because the source address is clear from the disassembly. For asynchronous events, however, the 
source address cannot be inferred from the source, and hence a FUP will be sent. Table 31-24 illustrates cases 

where FUPs are sent, and which IP can be expected in those cases. 


Table 31-24. FUP Cases and IP Payload 


Event 


Flow Update IP 


Comment 


External Interrupt, NMI/SMI, Traps, 
Machine Check (trap-like), INIT/SIPI 


Address of next instruction (Next IP) that 
would have been executed 


Functionally, this matches the LBR FROM field 
value and also the EIP value which is saved onto 
the stack. 


Exceptions/Faults, Machine check 
(fault-like) 


Address of the instruction which took the 
exception/fault (Current IP) 


This matches the similar functionality of LBR 
FROM field value and also the EIP value which is 
saved onto the stack. 


Software Interrupt 


Address of the software interrupt instruction 
(Current IP) 


This matches the similar functionality of LBR 
FROM field value, but does not match the EIP 
value which is saved onto the stack (Next 
Linear Instruction Pointer - NLIP). 


EENTER, EEXIT, ERESUME, Enclave 
Exiting Event (EEE), AEX! 


XACQUIRE 


Current IP of the instruction 


Address of the X* instruction 


This matches the LBR FROM field value and also 
the EIP value which is saved onto the stack. 


XRELEASE, XBEGIN, XEND, 
XABORT, other transactional abort 


Current IP 


#SMI 


IP that is saved into SMRAM 


WRMSR that clears TraceEn, PSB+ 


Current IP 


NOTES: 


1. Information on EENTER, EEXIT, ERESUME, EEE, Asynchronous Enclave eXit (AEX) can be found in the Inte/® 


64 and IA-32 Architectures Software Developer’s Manual, Volume 3D. 


On a canonical fault due to sequentially fetching an instruction in non-canonical space (as opposed to jumping to 
non-canonical space), the IP of the fault (and thus the payload of the FUP) will be a non-canonical address. This is 
consistent with what is pushed on the stack for such faulting cases. 
If there are post-commit task switch faults, the IP value of the FUP will be the original IP when the task switch 

started. This is the same value as would be seen in the LBR_FROM field. But it is a different value as is saved on the 


stack or VMCS. 
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31.4.2.7. Paging Information (PIP) Packet 


Table 31-25. PIP Packet Definition 


Name 


Paging Information (PIP) Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 0 0 0 1 0 
1 0 1 0 0 0 0 1 1 
2 CR3[11:5] or O RSVD/NR 
3 CR3[19:12] 
4 CR3[27:20] 
5 CR3[35:28] 
6 CR3[43:36] 
7 CR3[51:44] 
Dependencies | TriggerEn && ContextEn && Generation MOV CR3, Task switch, INIT, SIPI, PSB+, VM exit, 
IA32_RTIT_CTL.OS Scenario VM entry 
Description The CR3 payload shown includes only the address portion of the CR3 value. For PAE paging, CR3[11:5] are thus 
included. For other paging modes (32-bit and 4-level paging'), these bits are 0. 
This packet holds the CR3 address value. It will be generated on operations that modify CR3: 
«= MOV CR3 operation 
" Task Switch 
«INIT and SIPI 
= VM exit, if “conceal VMX from PT” VM-exit control is O (see Section 31.5.1) 
= VM entry, if “conceal VMX from PT” VM-entry control is 0 
PIPs are not generated, despite changes to CR3, on SMI and RSM. This is due to the special behavior on these oper- 
ations, see Section 31.2.8.3 for details. Note that, for some cases of task switch where CR3 is not modified, no PIP 
will be produced. 
The purpose of the PIP is to indicate to the decoder which application is running, so that it can apply the proper 
binaries to the linear addresses that are being traced. 
The PIP packet contains the new CR3 value when CR3 is written. 
PIPs generated by VM entries set the NR bit. PIPs generated in VMX non-root operation set the NR bit if the “con- 
ceal VMX from PT” VM-execution control is 0 (see Section 31.5.1). All other PIPs clear the NR bit. 
Application The purpose of the PIP packet is to help the decoder uniquely identify what software is running at any given time. 
When a PIP is encountered, a decoder should do the following: 
1) If there was a prior unbound FUP (that is, a FUP not preceded by a packet such as MODE.TSX that consumes it, 
and it hence pairs with a TIP that has not yet been seen), then this PIP is part of a compound packet event (Section 
31.4.1). Find the ending TIP and apply the new CR3/NR values to the TIP payload IP. 
2) Otherwise, look for the next MOV CR3, far branch, or VMRESUME/VMLAUNCH in the disassembly, and apply the 
new CR3 to the next (or target) IP. 
For examples of the packets generated by these flows, see Section 31.7. 
NOTES: 


1. Earlier versions of this manual used the term “IA-32e paging” to identify 4-level paging. 
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31.4.2.8 MODE Packets 


MODE packets keep the decoder informed of various processor modes about which it needs to know in order to 
properly manage the packet output, or to properly disassemble the associated binaries. MODE packets include a 
header and a mode byte, as shown below. 


Table 31-26. General Form of MODE Packets 


7 5 4 3} 2 1 0 
0 1 0 1 1 0 0 1 
1 Leaf ID Mode 


The MODE Leaf ID indicates which set of mode bits are held in the lower bits. 


MODE.Exec Packet 
Table 31-27. MODE.Exec Packet Definition 
Name MODE.Exec Packet 
Packet Format 
7 6 5 4 3) 2 1 0 
0 1 0 0 1 1 0 0 1 
1 0 0 0 0 0 0 cs.D (CS.L & LMA) 


Dependencies 


PacketEn Generation 


Scenario 


Far branch, interrupt, exception, VM exit, and VM entry, if the mode changes. 
PSB+, and any scenario that can generate a TIP.PGE, such that the mode may have 
changed since the last MODE.Exec. 


Description 


Indicates whether software is in 16, 32, or 64-bit mode, by providing the CS.D and (CS.L & IA32_EFER.LMA) values. 
Essential for the decoder to properly disassemble the associated binary. 


CS.D 


= 


CS.L & IA32_EFER.LMA) Addressing Mode 
N/A 

64-bit mode 
32-bit mode 


16-bit mode 


oO; =| Oo] — 


o|lo!l-|- 


MODE.Exec is sent at the time of a mode change, if PacketEn=1 at the time, or when tracing resumes, if necessary. 
In the former case, the MODE.Exec packet is generated along with other packets that result from the far transfer 
operation that changes the mode. In cases where the mode changes while PacketEn=0, the processor will send out 
a MODE.Exec along with the TIP.PGE when tracing resumes. The processor may opt to suppress the MODE.Exec 
when tracing resumes if the mode matches that from the last MODE.Exec packet, if there was no PSB in between. 


Application 


MODE.Exec always immediately precedes a TIP or TIP.PGE. The mode change applies to the IP address in the payload 


of the next TIP or TIP.PGE. 
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MODE.TSX Packet 
Table 31-28. MODE.TSX Packet Definition 
Name MODE.TSX Packet 
Packet Format 
7 6 5 4 3 2 1 0 

0 1 0 0 1 1 0 0 1 

1 0 0 1 0 0 0 TXAbort | InTX 
Dependencies | TriggerEn and ContextEn Generation XBEGIN, XEND, XABORT, XACQUIRE, XRELEASE, if InTX 

Scenario changes, Asynchronous TSX Abort, PSB+ 

Description Indicates when a TSX transaction (either HLE or RTM) begins, commits, or aborts. Instructions executed transaction- 


ally will be “rolled back” if the transaction is aborted. 


TXAbort InTX Implication 
1 1 N/A 


0 1 Transaction begins, or executing transactionally 
1 0 Transaction aborted 
0 0 Transaction committed, or not executing transactionally 


Application If PacketEn=1, MODE.TSX always immediately precedes a FUP. If the TXAbort bit is zero, then the mode change 
applies to the IP address in the payload of the FUP. If TXAbort=1, then the FUP will be followed by a TIP, and the 
mode change will apply to the IP address in the payload of the TIP. 

MODE.TSX packets may be generated when PacketEn=0, due to FilterEn=0. In this case, only the last MODE.TSX 
generated before TIP.PGE need be applied. 
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31.4.2.9 TraceStop Packet 


Table 31-29. TraceStop Packet Definition 


Name 


TraceStop Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 1 0 0 0 0 0) 1 1 
Dependencies | TriggerEn && ContextEn Generation Taken branch with target in TraceStop IP region, MOV CR3 in TraceS- 


Scenario top IP region, or WRMSR that sets TraceEn in TraceStop IP region. 


Description 


Indicates when software has entered a user-configured TraceStop region. 

When the IP matches a TraceStop range while ContextEn and Trigger€n are set, a TraceStop action occurs. This dis- 
ables tracing by setting IA32_RTIT_STATUS.Stopped, thereby clearing TriggerEn, and causes a TraceStop 

packet to be generated. 

The TraceStop action also forces FilterEn to 0. Note that TraceStop may not force a flush of internally buffered 
packets, and thus trace packet generation should still be manually disabled by clearing IA32_RTIT_CTL.TraceEn 
before examining output. See Section 31.2.4.3 for more details. 


Application 


If TraceStop follows a TIP.PGD (before the next TIP.PGE), then it was triggered either by the instruction that cleared 
PacketEn, or it was triggered by some later instruction that executed while FilterEn=0. In either case, the TraceStop 
can be applied at the IP of the TIPPGD (if any). 

If TraceStop follows a TIP.PGE (before the next TIP.PGD), it should be applied at the last known IP. 


31.4.2.10 Core:Bus Ratio (CBR) Packet 


Table 31-30. CBR Packet Definition 


Name Core:Bus Ratio (CBR) Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 0 0 0 0 0 0 1 1 
2 Core:Bus Ratio 
3 Reserved 
Dependencies | TriggerEn Generation After any frequency change, on C-state wake up, PSB+, and after 


Description 


Scenario enabling trace packet generation. 


Indicates the core:bus ratio of the processor core. Useful for correlating wall-clock time and cycle time. 


Application 


The CBR packet indicates the point in the trace when a frequency transition has occurred. On some implementa- 
tions, software execution will continue during transitions to a new frequency, while on others software execution 
ceases during frequency transitions. There is not a precise IP provided, to which to bind the CBR packet. 
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31.4.2.11 Timestamp Counter (TSC) Packet 


Table 31-31. TSC Packet Definition 


Name 


Timestamp Counter (TSC) Packet 


Packet Format 


Dependencies 


7 


6 


0 


) 


SW TSC[7:0] 


SW TSC[15:8] 


SW TSC[23:16] 


SW TSC[31:24] 


SW TSC[39:32] 


SW TSC[47:40] 


NJ] @]} ui} B&B} Wu] Poy =| © 


SW TSC[55:48] 


IA32_RTIT_CTL.TSCEn && | Generation 
Scenario 


TriggerEn 


Sent after any event that causes the processor clocks or Intel PT timing 
packets (such as MTC or CYC) to stop, This may include P-state changes, 
wake from C-state, or clock modulation. Also on transition of TraceEn 
from 0 to 1. 


Description When enabled by software, a TSC packet provides the lower 7 bytes of the current TSC value, as returned by the 
RDTSC instruction. This may be useful for tracking wall-clock time, and synchronizing the packets in the log with 
other timestamped logs. 

Application TSC packet provides a wall-clock proxy of the event which generated it (packet generation enable, sleep state wake, 


etc). In all cases, TSC does not precisely indicate the time of any control flow packets; however, all preceding packets 
represent instructions that executed before the indicated TSC time, and all subsequent packets represent instruc- 
tions that executed after it. There is not a precise IP to which to bind the TSC packet. 
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31.4.2.12 Mini Time Counter (MTC) Packet 


Table 31-32. MTC Packet Definition 


Name 


Mini time Counter (MTC) Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 1 0 1 1 0 0 1 
1 CTCIN+7:N] 


Dependencies 


IA32_RTIT_CTL.MTCEn && | Generation Periodic, based on the core crystal clock, or Always Running Timer 
TriggerEn Scenario (ART). 


Description 


When enabled by software, an MTC packet provides a periodic indication of wall-clock time. The 8-bit CTC (Common 
Timestamp Copy) payload value is set to (ART >> N) & FFH. The frequency of the ART is related to the Maximum 
Non-Turbo frequency, and the ratio can be determined from CPUID leaf 15H, as described in Section 31.8.3. 
Software can select the threshold N, which determines the MTC frequency by setting the IA32_RTIT_CTL.MTCFreq 
field (see Section 31.2.7.2) to a supported value using the lookup enumerated by CPUID (see Section 31.3.1). 

See Section 31.8.3 for details on how to use the MTC payload to track TSC time. 

MTC provides 8 bits from the ART, starting with the bit selected by MTCFreq to dictate the frequency of the packet. 
Whenever that 8-bit range being watched changes, an MTC packet will be sent out with the new value of that 8-bit 
range. This allows the decoder to keep track of how much wall-clock time has elapsed since the last TSC packet was 
sent, by keeping track of how many MTC packets were sent and what their value was. The decoder can infer the 
truncated bits, CTC[N-1:0], are O at the time of the MTC packet. 

There are cases in which MTC packet can be dropped, due to overflow or other micro-architectural conditions. The 
decoder should be able to recover from such cases by checking the 8-bit payload of the next MTC packet, to deter- 
mine how many MTC packets were dropped. It is not expected that >256 consecutive MTC packets should ever be 
dropped. 


Application 


MTC does not precisely indicate the time of any other packet, nor does it bind to any IP. However, all preceding pack- 
ets represent instructions or events that executed before the indicated ART time, and all subsequent packets repre- 
sent instructions that executed after, or at the same time as, the ART time. 
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31.4.2.13 TSC/MTC Alignment (TMA) Packet 


Table 31-33. TMA Packet Definition 
Name TSC/MTC Alignment (TMA) Packet 


Packet Format 


7 6 5 4 3 2 1 0 

0 0 0 0 0 0 0 1 0 

1 0 1 1 1 0) 0 1 1 

2 CTC[7:0] 

3 CTC[15:8] 

4 Reserved 0 

5 FastCounter[7:0] 

6 Reserved FC[8] 
Dependencies | IA32_RTIT_CTL.MTCEn && Generation Sce- Sent with any TSC packet. 

IA32_RTIT_CTL.TSCEn && TriggerEn | nario 

Description The TMA packet serves to provide the information needed to allow the decoder to correlate MTC packets with TSC 


packets. With this packet, when a MTC packet is encountered, the decoder can determine how many timestamp 
counter ticks have passed since the last TSC or MTC packet. See Section 31.8.3.2 for details on how to make this cal- 
culation. 


Application TMA is always sent immediately following a TSC packet, and the payload values are consistent with the TSC payload 
value. Thus the application of TMA matches that of TSC. 
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31.4.2.14 Cycle Count (CYC) Packet 


Table 31-34. Cycle Count Packet Definition 


Name Cycle Count (CYC) Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 Cycle Counter[4:0] Exp 1 1 
1 Cycle Counter[11:5] Exp 
2 Cycle Counter[18:1 2] Exp 
.. (if Exp = 1 in the previous byte) 


Dependencies 


IA32_RTIT_CTL.CYCEn && | Generation Sce- | Can be sent at any time, though a maximum of one CYC packet is 
TriggerEn nario sent per core clock cycle. See Section 31.3.6 for CYC-eligible packets. 


Description 


The Cycle Counter field increments at the same rate as the processor core clock ticks, but with a variable length for- 
mat (using a trailing EXP bit field) and a range-capped byte length. 

If the CYC value is less than 32, a 1-byte CYC will be generated, with Exp=0. If the CYC value is between 32 and 
4095 inclusive, a 2-byte CYC will be generated, with byte 0 Exp=1 and byte 1 Exp=0. And so on. 

CYC provides the number of core clocks that have passed since the last CYC packet. CYC can be configured to be 
sent in every cycle in which an eligible packet is generated, or software can opt to use a threshold to limit the num- 
ber of CYC packets, at the expense of some precision. These settings are configured using the 
IA32_RTIT_CTL.CycThresh field (see Section 31.2.7.2). For details on Cycle-Accurate Mode, IPC calculation, etc, see 
Section 31.3.6. 

When CycThresh=0, and hence no threshold is in use, then a CYC packet will be generated in any cycle in which any 
CYC-eligible packet is generated. The CYC packet will precede the other packets generated in the cycle, and provides 
the precise cycle time of the packets that follow. 

In addition to these CYC packets generated with other packets, CYC packets can be sent stand-alone. These packets 
serve simply to update the decoder with the number of cycles passed, and are used to ensure that a wrap of the 
processor's internal cycle counter doesn’t cause cycle information to be lost. These stand-alone CYC packets do not 
indicate the cycle time of any other packet or operation, and will be followed by another CYC packet before any 
other CYC-eligible packet is seen. 

When CycThresh>0, CYC packets are generated only after a minimum number of cycles have passed since the last 
CYC packet. Once this threshold has passed, the behavior above resumes, where CYC will either be sent in the next 
cycle that produces other CYC-eligible packets, or could be sent stand-alone. 

When using CYC thresholds, only the cycle time of the operation (instruction or event) that generates the CYC 
packet is truly known. Other operations simply have their execution time bounded: they completed at or after the 
last CYC time, and before the next CYC time. 


Application 


CYC provides the offset cycle time (since the last CYC packet) for the CYC-eligible packet that follows. If another CYC 
is encountered before the next CYC-eligible packet, the cycle values should be accumulated and applied to the next 
CYC-eligible packet. 

If a CYC packet is generated by a TNT, note that the cycle time provided by the CYC packet applies to the first 
branch in the TNT packet. 
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31.4.2.15 VMCS Packet 


Table 31-35. VMCS Packet Definition 


Name 


VMCS Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 
1 1 0 0 1 0) 0 0 
VMCS pointer [19:12] 
VMCS pointer [27:20] 
VMCS pointer [35:28] 
VMCS pointer [43:36] 
VMCS pointer [51:44] 


(=) 
[o) 
(=) 
os 
(=) 


DM] Wy BY] Wy] rN] —| © 


Dependencies 


TriggerEn && ContextEn; | Generation Scenario | Generated on successful VMPTRLD, and optionally on PSB+, SMM 
Also in VMX operation. VM exits, and VM entries that return from SMM (see Section 31- 
51). 


Description The VMCS packet provides a VMCS pointer for a decoder to determine the transition of code contexts: 
® Onasuccessful VMPTRLD (i.e.,a VMPTRLD that doesn't fault, fail, or VM exit), the VMCS packet contains the 
logical processor's VMCS pointer established by VMPTRLD (for subsequent execution of a VM guest context). 
e ~AnSMM VM exit loads the logical processor's VMCS pointer with the SMM-transfer VMCS pointer. If the “conceal 
VMX from PT” VM-exit control is 0 (see Section 31.5.1), a VMCS packet provides this pointer. See Section 31.6 on 
tracing inside and outside STM. 
e =AVMentry that returns from SMM loads the logical processor's VMCS pointer from a field in the SMM-transfer 
VMCS. If the “conceal VMX from PT” VM-entry control is 0, a VMCS packet provides this pointer. Whether the 
VM entry is to VMX root operation or VMX non-root operation is indicated by the PIP.NR bit. 
A VMCS packet generated before a VMCS pointer has been loaded, or after the VMCS pointer has been cleared will 
set all 64 bits in the VMCS pointer field. 
VMCS packets will not be seen on processors with IA32_VMX_MISC[bit 14]=0, as these processors do not allow 
TraceEn to be set in VMX operation. 
Application The purpose of the VMCS packet is to help the decoder uniquely identify changes in the executing software context 


in situations that CR3 may not be unique. 

When a VMCS packet is encountered, a decoder should do the following: 

«If there was a prior unbound FUP (that is, a FUP not preceded by a packet such as MODE.TSX that consumes it, and 
it hence pairs with a TIP that has not yet been seen), then this VMCS is part of a compound packet event (Section 
31.4.1). Find the ending TIP and apply the new VMCS base pointer value to the TIP payload IP. 

* Otherwise, look for the next VMPTRLD, VMRESUME, or VMLAUNCH in the disassembly, and apply the new VMCS 
base pointer on the next VM entry. 

For examples of the packets generated by these flows, see Section 31.7. 
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31.4.2.16 Overflow (OVF) Packet 


Table 31-36. OVF Packet Definition 


Name Overflow (OVF) Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 1 1 1 1 0 0 1 1 
Dependencies | TriggerEn Generation On resolution of internal buffer overflow 
Scenario 
Description OVF simply indicates to the decoder that an internal buffer overflow occurred, and packets were likely lost. If 
BranchEN= 1, OVF is followed by a FUP or TIP.PGE which will provide the IP at which packet generation resumes. See 
Section 31.3.8. 
Application When an OVF packet is encountered, the decoder should skip to the IP given in the subsequent FUP or TIP.PGE. The 


cycle counter for the CYC packet will be reset at the time the OVF packet is sent. 

Software should reset its call stack depth on overflow, since no RET compression is allowed across an overflow. Sim- 
ilarly, any IP compression that follows the OVF is guaranteed to use as a reference LastIP the IP payload of an IP 
packet that preceded the overflow. 


31.4.2.17 Packet Stream Boundary (PSB) Packet 


Table 31-37. PSB Packet Definition 


Name 


Packet Stream Boundary (PSB) Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 1 0 0 0 0 0 1 0 
2 0 0 0 0 0 0 1 0 
3 1 0 0 0 0 0 1 0 
4 0 0 0 0 0 0 1 0 
5 1 0 0 0 0 0 1 0 
6 0 0 0 0 0 0 1 0 
7 1 0 0 0 0 0 1 0 
8 0 0 0 0 0 0 1 0 
s] 1 0 0 0 0 0 1 0 
10 0 0 0 0 0 0 1 0 
11 1 0 0 0 0 0 1 0 
12 0 0 0 0 0 0 1 0 
13 1 0 0 0 0 0 1 0 
14 0 0 0 0 0 0 1 0 
15 1 0 0 0 0 0 1 0 
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Table 31-37. PSB Packet Definition (Contd.) 


Dependencies 


Periodic, based on the number of output bytes generated while tracing. PSB is sent 
when IA32_RTIT_STATUS.PacketByteCnt=0, and each time it crosses the software 
selected threshold after that. May be sent for other micro-architectural conditions 
as well. 


Generation 
Scenario 


TriggerEn 


Description 


PSB is a unique pattern in the packet output log, and hence serves as a sync point for the decoder. It is a pattern 
that the decoder can search for in order to get aligned on packet boundaries. This packet is periodic, based on the 
number of output bytes, as indicated by [A32_RTIT_STATUS.PacketByteCnt. The period is chosen by software, via 
IA32_RTIT_CTL.PSBFreq (see Section 31.2.7.2). Note, however, that the PSB period is not precise, it simply reflects 
the average number of output bytes that should pass between PSBs. The processor will make a best effort to 
insert PSB as quickly after the selected threshold is reached as possible. The processor also may send extra 

PSB packets for some micro-architectural conditions. 

PSB also serves as the leading packet for a set of “status-only” packets collectively known as PSB+ (Section 31.3.7). 


Application 


When a PSB is seen, the decoder should interpret all following packets as “status only”, until either a PSBEND or 
OVF packet is encountered. “Status only” implies that the binding and ordering rules to which these packets nor- 
mally adhere are ignored, and the state they carry can instead be applied to the IP payload in the FUP packet that is 
included. 


31.4.2.18 PSBEND Packet 


Table 31-38. PSBEND Packet Definition 


Name PSBEND Packet 
Packet Format 
7 6 5 4 3) 1 0 

0 0 0 0 1 0 

1 0 0 1 0 0 0 1 1 
Dependencies | TriggerEn Generation Always follows PSB packet, separated by PSB+ packets 

Scenario 

Description PSBEND is simply a terminator for the series of “status only” (PSB+) packets that follow PSB (Section 31.3.7). 
Application When a PSBEND packet is seen, the decoder should cease to treat packets as “status only”. 
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31.4.2.19 Maintenance (MNT) Packet 


Table 31-39. MNT Packet Definition 


Name 


Maintenance (MNT) Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 1 1 0 0 0 0 1 1 
2 1 0 0 0 1 0 0 0 
3 Payload[7:0] 
4 Payload[1 5:8] 
5 Payload[23:16] 
6 Payload[31:24] 
7 Payload[39:32] 
8 Payload[47:40] 
9 Payload[55:48] 
10 Payload[63:56] 
Dependencies TriggerEn Generation Sce- | Implementation specific. 
nario 
Description This packet is generated by hardware, the payload meaning is model-specific. 
Application Unless a decoder has been extended for a particular family/model/stepping to interpret MNT packet payloads, this 


packet should simply be ignored. It does not bind to any IP. 


31.4.2.20 PAD Packet 


Table 31-40. PAD Packet Definition 


Name PAD Packet 
Packet Format 
7 6 5) 4 3 2 1 0 
0 0 0 0 0) 0 0 0 
Dependencies | TriggerEn Generation Implementation specific 
Scenario 
Description PAD is simply a NOP packet. Processor implementations may choose to add pad packets to improve packet align- 
ment or for implementation-specific reasons. 
Application Ignore PAD packets. 
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31.4.2.21 PTWRITE (PTW) Packet 


Table 31-41. PTW Packet Definition 


Name 


PTW Packet 


Packet Format 


7 


6 


5 4 


0 


0 


0 0 


IP 


PayloadBytes 


is) 


Payload[7:0] 


Payload[15:8] 


Payload[23:16] 


Payload[31:24] 


Payload[39:32] 


Payload[47:40] 


Payload[55:48] 
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Payload[63:56] 


The PayloadBytes field indicates the number of bytes of payload that follow the header bytes. Encodings are as fol- 


lows: 
PayloadBytes Bytes of Payload 
‘00 4 
‘01 8 
‘10 Reserved 
‘11 Reserved 


IP bit indicates if a FUP, whose payload will be the IP of the PTWRITE instruction, will follow. 


Dependencies 


TriggerEn & ContextEn & FilterEn | Generation 


PTWRITE Instruction 


PTW.IP=1. 


& PTWEn Scenario 

Description Contains the value held in the PTWRITE operand. 
This packet is CYC-eligible, and hence will generate a CYC packet if IA32_RTIT_CTL.CYCEn=1 and any CYC Threshold 
has been reached. 

Application Binds to the associated PTWRITE instruction. The IP of the PTWRITE will be provided by a following FUP, when 
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31.4.2.22 Execution Stop (EXSTOP) Packet 


Table 31-42. EXSTOP Packet Definition 


Name EXSTOP Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 0 0 0 0 0 1 0 
1 IP 1 1 0 0 0 1 0 
IP bit indicates if a FUP will follow. 
Dependencies | TriggerEn & Pwr€vtEn Generation C-state entry, P-state change, or other processor clock power- 
Scenario down. Includes : 
« Entry to C-state deeper than CO.0 
= TM1/2 
= STPCLK# 
« Frequency change due to |[A32_CLOCK_MODULATION, Turbo 


Description 


If a FUP follows EXSTOP (hence IP bit set), the EXSTOP can be bound to the FUP IP. Otherwise the IP is not known. 


Application 


This packet indicates that software execution has stopped due to processor clock powerdown. Later packets will 
indicate when execution resumes. 

If EXSTOP is generated while ContextEn is set, the IP bit will be set, and EXSTOP will be followed by a FUP packet 
containing the IP at which execution stopped. More precisely, this will be the IP of the oldest instruction that has 
not yet completed. 

This packet is CYC-eligible, and hence will generate a CYC packet if IA32_RTIT_CTL.CYCEn=1 and any CYC Threshold 
has been reached. 


Time of powerdown can be inferred from the preceding CYC, if CYCEn=1. Combined with the TSC at the time of 
wake (if TSCEn=1), this can be used to determine the duration of the powerdown. 
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31.4.2.23  MWAIT Packet 


Table 31-43. MWAIT Packet Definition 


Name 


MWAIT Packet 


Packet Format 


7 6 5 4 3 2 1 0 
0 0 0 0 0 0 1 0 
1 1 0 0 0) 0 1 0 
MWAIT Hints[7:0] 

Reserved 


Reserved 


Reserved 
Reserved EXT[1:0] 
Reserved 


Reserved 
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Reserved 


Dependencies 


TriggerEn & PwrEvtEn & Contex- | Generation MWAIT, UMWAIT, or TPAUSE instructions, or I/O redirection to 
tEn Scenario MWAIT, that complete without fault or VMexit. 


Description 


Indicates that an MWAIT operation to C-state deeper than CO.0 completed. The MWAIT hints and extensions passed 
in by software are exposed in the payload. For UMWAIT and TPAUSE, the EXT field holds the input register value 
that determines the optimized state requested. 

For entry to some highly optimized CO sub-C-states, such as C0.1, no MWAIT packet is generated. 

This packet is CYC-eligible, and hence will generate a CYC packet if IA32_RTIT_CTL.CYCEn=1 and any CYC Threshold 
has been reached. 


Application 
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31.4.2.24 Power Entry (PWRE) Packet 


Table 31-44. PWRE Packet Definition 


Name PWRE Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 0 0 0 0 0 1 0 
1 0 0 1 0 0 0 1 0 
2 HW Reserved 
3 Resolved Thread C-State Resolved Thread Sub C-State 
Dependencies | TriggerEn & Pwr€vtEn Generation Transition to a C-state deeper than CO0.0. 
Scenario 
Description Indicates processor entry to the resolved thread C-state and sub C-state indicated. The processor will remain in this 
C-state until either another PWRE indicates the processor has moved to a C-state deeper than CO.0, or a PWRX 
packet indicates a return to CO.0. 
For entry to some highly optimized CO sub-C-states, such as CO.1, no PWRE packet is generated. 
Note that some CPUs may allow MWAIT to request a deeper C-state than is supported by the core. These deeper C- 
states may have platform-level implications that differentiate them. However, the PWRE packet will provide only 
the resolved thread C-state, which will not exceed that supported by the core. 
If the C-state entry was initiated by hardware, rather than a direct software request (such as MWAIT, UMWAIT, 
TPAUSE, HLT, or shutdown), the HW bit will be set to indicate this. Hardware Duty Cycling (see Section 14.5, “Hard- 
ware Duty Cycling (HDC)” in the Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 38) is an 
example of such a case. 
Application When transitioning from CO.0 to a deeper C-state, the PWRE packet will be followed by an EXSTOP. If that EXSTOP 


packet has the IP bit set, then the following FUP will provide the IP at which the C-state entry occurred. Subsequent 
PWRE packets generated before the next PWRX should bind to the same IP. 
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31.4.2.25 Power Exit (PWRX) Packet 


Table 31-45. PWRX Packet Definition 


Name PWRX Packet 
Packet Format 
7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 1 0 1 0 0) 0 1 0 
2 Last Core C-State Deepest Core C-State 
3) Reserved Wake Reason 
4 Reserved 
5 Reserved 
6 Reserved 
Dependencies | TriggerEn & PwrEvtEn Generation Transition from a C-state deeper than C0.0 to CO. 
Scenario 


Description 


Application 


Indicates processor return to thread CO from a C-state deeper than C0.0. 

For return from some highly optimized CO sub-C-states, such as CO.1, no PWRX packet is generated. 

The Last Core C-State field provides the MWAIT encoding for the core C-state at the time of the wake. The Deepest 
Core C-State provides the MWAIT encoding for the deepest core C-state achieved during the sleep session, or since 
leaving thread CO. MWAIT encodings for C-states can be found in Table 4-11 in the Intel® 64 and IA-32 Architec- 
tures Software Developer's Manual, Volume 2B. Note that these values reflect only the core C-state, and hence will 
not exceed the maximum supported core C-state, even if deeper C-states can be requested. 

The Wake Reason field is one-hot, encoded as follows: 


Bit Field Meaning 

0 Interrupt Wake due to external interrupt received. 

1 Timer Deadline Wake due to timer expiration, such as 
UMWAIT/TPAUSE TSC-quanta. 

2 Store to Monitored Address | Wake due to store to monitored address. 

3 HW Wake Wake due to hardware autonomous condition, 
such as HDC. 


PWRX will always apply to the same IP as the PWRE. The time of wake can be discerned from (optional) timing pack- 
ets that precede PWRX. 
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Table 31-46. Block Begin Packet Definition 
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Name BBP 
Packet Format 
7 6 5 4 3 2 1 0 
0 0 0 0 0 0 0 1 0 
1 0 1 1 0 0 0 1 1 
2 SZ Reserved Type[4:0] 
Dependencies | TriggerEn Generation PEBS event, if IA32_PEBS_ENABLE.OUTPUT=1. 
Scenario 
Description This packet indicates the beginning of a block of packets which are collectively tied to a single event or instruction. 
The size of the block item payloads within this block is provided by the Size (SZ) bit: 
SZ=0: 8-byte block items 
SZ=1: 4-byte block items 
The meaning of the BIP payloads is provided by the Type field: 
BBP.Type Block name 
0x00 Reserved 
0x01 General-Purpose Registers 
0x02..0x03 Reserved 
0x04 PEBS Basic 
0x05 PEBS Memory 
0x06..0x07 Reserved 
0x08 LBR Block 0 
0x09 LBR Block 1 
Ox0A LBR Block 2 
Ox0OB..0x0F Reserved 
0x10 XMM Registers 
0x11..0x1F Reserved 
Application A BBP will always be followed by a Block End Packet (BEP), and when the block is generated while ContextEn=1 


that BEP will have IP=1 and be followed by a FUP that provides the IP to which the block should be bound. Note 
that, in addition to BEP, a block can be terminated by a BBP (indicating the start of a new block) or an OVF packet. 
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31.4.2.27 Block Item Packet (BIP) 


Table 31-47. Block Item Packet Definition 


Name BIP 


Packet Format | If the preceding BBP.SZ=0: 


7 6 5 4 3 2 1 0 
0 ID[5:0] 1 0 0 
1 Payload[7:0] 
2 Payload[15:8] 
3 Payload[23:16] 
4 Payload[31:24] 
5 Payload[39:32] 
6 Payload[47:40] 
7 Payload[55:48] 
8 Payload[63:56] 

If the preceding BBP.SZ=1: 

7 6 5 4 3 2 1 0 
0 ID[5:0] 1 0 0 
1 Payload[7:0] 
2 Payload[15:8] 
3} Payload[23:16] 
4 Payload[31:24] 

Dependencies | TriggerEn Generation See BBP. 
Scenario 
Description The size of the BIP payload is determined by the Size field in the preceding BBP packet. 


The BIP header provides the ID value that, when combined with the Type field from the preceding BBP, uniquely 
identifies the state value held in the BIP payload. See Table 31-48 below for the complete list. 


Application See BBP. 


BIP State Value Encodings 


The table below provides the encoding values for all defined block items. State items that are larger than 8 bytes, 
such as XMM register values, are broken into multiple 8-byte components. BIP packets with Size=1 (4 byte 
payload) will provide only the lower 4 bytes of the associated state value. 


Table 31-48. BIP Encodings 


BBP.Type BIP.ID State Value 
General-Purpose Registers 
0x01 0x00 R/EFLAGS 
0x01 0x01 R/EIP 
0x01 0x02 R/EAX 
0x01 0x03 R/ECX 
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BBP.Type BIP.ID State Value 
0x01 0x04 R/EDX 
0x01 0x05 R/EBX 
0x01 0x06 R/ESP 
0x01 0x07 R/EBP 
0x01 0x08 R/ESI 
0x01 0x09 R/EDI 
0x01 Ox0A R8 
0x01 0x0B RO 
0x01 Ox0C R10 
0x01 0x0D R11 
0x01 Ox0E R12 
0x01 OxOF R13 
0x01 0x10 R14 
0x01 0x11 R15 
PEBS Basic Info (Section 18.9.2.2.1) 
0x04 0x00 Instruction Pointer 
0x04 0x01 Applicable Counters 
0x04 0x02 Timestamp 
PEBS Memory Info (Section 18.9.2.2.2) 
0x05 0x00 MemAccessAddress 
0x05 0x01 MemAuxinfo 
0x05 0x02 MemAccessLatency 
0x05 0x03 TSXAuxInfo 
LBR_O 
0x08 0x00 LBR[TOS-0]_FROM_IP 
0x08 0x01 LBR[TOS-O]_TO_IP 
0x08 0x02 LBR[TOS-O]_INFO 
0x08 0x03 LBR[TOS-1]_FROM_IP 
0x08 0x04 LBR[TOS-1]_TO_IP 
0x08 0x05 LBR[TOS-1]_INFO 
0x08 0x06 LBR[TOS-2]_FROM_IP 
0x08 0x07 LBR[TOS-2]_TO_IP 
0x08 0x08 LBR[TOS-2]_INFO 
0x08 0x09 LBR[TOS-3]_FROM_IP 
0x08 Ox0A LBR[TOS-3]_TO_IP 
0x08 0x0B LBR[TOS-3]_INFO 
0x08 Ox0C LBR[TOS-4]_FROM_IP 
0x08 0x0D LBR[TOS-4]_TO_IP 
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Table 31-48. BIP Encodings 


BBP.Type BIP.ID State Value 
0x08 OxOE LBR[TOS-4]_INFO 
0x08 OxOF LBR[TOS-5]_FROM_IP 
0x08 0x10 LBR[TOS-5]_TO_IP 
0x08 0x11 LBR[TOS-5]_INFO 
0x08 0x12 LBR[TOS-6]_FROM_IP 
0x08 0x13 LBR[TOS-6]_TO_IP 
0x08 0x14 LBR[TOS-6]_INFO 
0x08 0x15 LBR[TOS-7]_FROM_IP 
0x08 0x16 LBR[TOS-7]_TO_IP 
0x08 0x17 LBR[TOS-7]_INFO 
0x08 0x18 LBR[TOS-8]_FROM_IP 
0x08 0x19 LBR[TOS-8]_TO_IP 
0x08 Ox1A LBR[TOS-8]_INFO 
0x08 0x1B LBR[TOS-9]_FROM_IP 
0x08 Ox1C LBR[TOS-9]_TO_IP 
0x08 0x1D LBR[TOS-9]_INFO 
0x08 Ox1E LBR[TOS-10]_FROM_IP 
0x08 Ox1F LBR[TOS-10]_TO_IP 

LBR_1 

0x09 0x00 LBR[TOS-10]_INFO 
0x09 0x01 LBR[TOS-11]_FROM_IP 
0x09 0x02 LBR[TOS-11]_TO_IP 
0x09 0x03 LBR[TOS-11]_INFO 
0x09 0x04 LBR[TOS-12]_ FROM_IP 
0x09 0x05 LBR[TOS-12]_TO_IP 
0x09 0x06 LBR[TOS-12]_INFO 
0x09 0x07 LBR[TOS-13]_FROM_IP 
0x09 0x08 LBR[TOS-13]_TO_IP 
0x09 0x09 LBR[TOS-13]_INFO 
0x09 Ox0A LBR[TOS-14]_FROM_IP 
0x09 0x0B LBR[TOS-14]_TO_IP 
0x09 Ox0C LBR[TOS-14]_INFO 
0x09 0x0D LBR[TOS-15]_ FROM_IP 
0x09 OxOE LBR[TOS-15]_TO_IP 
0x09 OxOF LBR[TOS-15]_INFO 
0x09 0x10 LBR[TOS-16]_FROM_IP 
0x09 0x11 LBR[TOS-16]_TO_IP 
0x09 0x12 LBR[TOS-16]_INFO 
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BBP.Type BIP.ID State Value 
0x09 0x13 LBR[TOS-17]_FROM_IP 
0x09 0x14 LBR[TOS-17]_TO_IP 
0x09 0x15 LBR[TOS-17]_INFO 
0x09 0x16 LBR[TOS-18]_FROM_IP 
0x09 0x17 LBR[TOS-18]_TO_IP 
0x09 0x18 LBR[TOS-18]_INFO 
0x09 0x19 LBR[TOS-19]_FROM_IP 
0x09 Ox1A LBR[TOS-19]_TO_IP 
0x09 0x1B LBR[TOS-19]_INFO 
0x09 Ox1C LBR[TOS-20]_FROM_IP 
0x09 0x1D LBR[TOS-20]_TO_IP 
0x09 Ox1E LBR[TOS-20]_INFO 
0x09 Ox1F LBR[TOS-21]_FROM_IP 

LBR_2 

Ox0A 0x00 LBR[TOS-21]_TO_IP 
Ox0A 0x01 LBR[TOS-21]_INFO 
Ox0A 0x02 LBR[TOS-22] FROM_IP 
Ox0A 0x03 LBR[TOS-22]_TO_IP 
Ox0A 0x04 LBR[TOS-22]_INFO 
Ox0A 0x05 LBR[TOS-23]_FROM_IP 
Ox0A 0x06 LBR[TOS-23]_TO_IP 
Ox0A 0x07 LBR[TOS-23]_INFO 
Ox0A 0x08 LBR[TOS-24]_ FROM_IP 
Ox0A 0x09 LBR[TOS-24]_TO_IP 
Ox0A Ox0A LBR[TOS-24]_INFO 
Ox0A Ox0B LBR[TOS-25]_ FROM_IP 
Ox0A Ox0C LBR[TOS-25]_TO_IP 
Ox0A 0x0D LBR[TOS-25]_INFO 
Ox0A Ox0E LBR[TOS-26]_FROM_IP 
Ox0A OxOF LBR[TOS-26]_TO_IP 
Ox0A 0x10 LBR[TOS-26]_INFO 
Ox0A 0x11 LBR[TOS-27]_FROM_IP 
Ox0A 0x12 LBR[TOS-27]_TO_IP 
Ox0A 0x13 LBR[TOS-27]_INFO 
Ox0A 0x14 LBR[TOS-28]_FROM_IP 
Ox0A 0x15 LBR[TOS-28]_TO_IP 
Ox0A 0x16 LBR[TOS-28]_INFO 
Ox0A 0x17 LBR[TOS-29]_FROM_IP 
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Table 31-48. BIP Encodings 


BBP.Type BIP.ID State Value 
Ox0A 0x18 LBR[TOS-29]_TO_IP 
Ox0A 0x19 LBR[TOS-29]_INFO 
Ox0A Ox1A LBR[TOS-30]_FROM_IP 
Ox0A 0x1B LBR[TOS-30]_TO_IP 
Ox0A Ox1C LBR[TOS-30]_INFO 
Ox0A 0x1D LBR[TOS-31]_FROM_IP 
OxOA Ox1E LBR[TOS-31]_TO_IP 
Ox0A Ox1F LBR[TOS-31]_INFO 
XMM Registers 
0x10 0x00 XMMO_QO 
0x10 0x01 XMMO_Q1 
0x10 0x02 XMM1_Q0 
0x10 0x03 XMM1_Q1 
0x10 0x04 XMM2_Q0 
0x10 0x05 XMM2_Q1 
0x10 0x06 XMM3_Q0 
0x10 0x07 XMM3_Q1 
0x10 0x08 XMM4_Q0 
0x10 0x09 XMM4_Q1 
0x10 Ox0A XMM5_QO 
0x10 0x0B XMM5_Q1 
0x10 Ox0C XMM6_QO 
0x10 0x0D XMM6_Q1 
0x10 OxOE XMM7_QO 
0x10 OxOF XMM7_Q1 
0x10 0x10 XMM8_QO 
0x10 0x11 XMM8_Q1 
0x10 0x12 XMM9_QO 
0x10 0x13 XMM9_Q1 
0x10 0x14 XMM10_Q0 
0x10 0x15 XMM10_Q1 
0x10 0x16 XMM11_Q0 
0x10 0x17 XMM11_Q1 
0x10 0x18 XMM12_Q0 
0x10 0x19 XMM12_Q1 
0x10 Ox1A XMM13_Q0 
0x10 0x1B XMM13_Q1 
0x10 Ox1C XMM14_Q0 
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Table 31-48. BIP Encodings 


BBP.Type BIP.ID State Value 
0x10 0x1D XMM14_Q1 
0x10 Ox1E XMM15_Q0 
0x10 Ox1F XMM15_Q1 


31.4.2.28 Block End Packet (BEP) 


Table 31-49. Block End Packet Definition 


Name BEP 
Packet Format 
7 6 5 4 3 2 1 0 

0 0 0 0 0 0 0 1 0 

1 IP 0 1 1 0 0 1 1 
Dependencies | TriggerEn Generation See BBP. 

Scenario 

Description Indicates the end of a packet block. The IP bit indicates if a FUP will follow, and will be set if ContextEn=1. 
Application The block, from initial BBP to the BEP, binds to the FUP IP, if IP=1, and consumes the FUP. 


31.5 TRACING IN VMX OPERATION 


On processors that IA32_VMX_MISC[bit 14] reports 1, TraceEn can be set in VMX operation. The VMM can 
configure specific VMX controls to control what virtualization-specific data is included within the trace packets (see 
Section 31.5.1 for details). The VMM can also configure the VMCS to limit tracing to non-root operation, or to trace 
across both root and non-root operation. The VMCS controls exist to simplify virtualization of Intel PT for guest use, 
including the “Clear IA32_RTIT_CTL” exit control (See Section 23.7.1), “Load IA32_RTIT_CTL” entry control (See 
Section 23.8.1), and “Intel PT uses guest physical addresses” execution control (See Section 24.5.3). 


For older processors that do not support these VMCS controls, the MSR-load areas used by VMX transitions can be 
employed by the VMM to restrict tracing to the desired context. See Section 31.5.2 for details. Tracing with SMM 
Transfer Monitor is described in Section 31.6. 


31.5.1 VMX-Specific Packets and VMCS Controls 


In all of the usages of VMX and Intel PT, a decoder in the host or VMM context can identify the occurrences of VMX 
transitions with the aid of VMX-specific packets. There are two kinds of packets relevant to VMX: 


° VMCS packet. The VMX transitions of individual VMs can be distinguished by a decoder using the VMCS- 
pointer field in a VMCS packet. A VMCS packet is sent on a successful execution of VMPTRLD, and its VMCS- 
pointer field stores the VMCS pointer loaded by that execution. See Section 31.4.2.15 for details. 


¢ The NR (non-root) bit in a PIP packet. Normally, the NR bit is set in any PIP packet generated in VMX non- 
root operation. In addition, PIP packets are generated with each VM entry and VM exit. Thus a transition of the 
NR bit from 0 to 1 indicates the occurrence of a VM entry, and a transition of 1 to 0 indicates the occurrence of 
a VM exit. 


There are VMX controls that a VMM can set to conceal some of this VMX-specific information (by suppressing its 
recording) and thereby prevent it from leaking across virtualization boundaries. There is one of these controls 
(each of which is called “conceal VMX from PT”) of each type of VMX control. 
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Table 31-50. VMX Controls For Intel Processor Trace 


Type of VMX Bit Value Behavior 
Control Position! 
Secondary 19 0) Each PIP generated in VM non-root operation will set the NR bit. 
processor-based PSB+ in VMX non-root operation will include the VMCS packet, to ensure that the decoder 
VM-execution knows which guest is currently in use. 
eae 1 Each PIP generated in VMX non-root operation will clear the NR bit. 
PSB+ in VMX non-root operation will not include the VMCS packet. 
V/M-exit control 24 0 Each VM exit generates a PIP in which the NR bit is clear. 
In addition, SMM VM exits generate VMCS packets. 
1 V/M exits do not generate PIPs, and no VMCS packets are generated on SMM VM exits. 


— 


7 0 Each VM entry generates a PIP in which the NR bit is set (except VM entries that return 
from SMM to VMX root operation). 


In addition, VM entries that return from SMM generate VMCS packets. 


1 \V/M entries do not generate PIPs, and no VMCS packets are generated on VM entries that 
return from SMM. 


V/M-entry control 


NOTES: 
1. These are the positions of the control bits in the relevant VMX control fields. 


The O-settings of these VMX controls enable all VMX-specific packet information. The scenarios that would use 
these default settings also do not require the VMM to use VMX MSR-load areas to enable and disable trace-packet 
generation across VMX transitions. 


If IA32_VMX_MISC[bit 14] reports 0, the 1-settings of the VMX controls in Table 31-50 are not supported, and 
VM entry will fail on any attempt to set them. 


31.5.2 Managing Trace Packet Generation Across VMX Transitions 


In tracing scenarios that collect packets for both VMX root operation and VMX non-root operation, a host executive 
can manage the MSRs associated with trace packet generation directly. The states of these MSRs need not be modi- 
fied across VMX transitions. 


For tracing scenarios that collect packets only within VMX root operation or only within VMX non-root operation, the 
VMM can toggle IA32_RTIT_CTL.TraceEn on VMxX transitions. 


31.5.2.1 System-Wide Tracing 


When a host or VMM configures Intel PT to collect trace packets of the entire system, it can leave the relevant VMX 
controls clear to allow VMX-specific packets to provide information across VMX transitions. 


The decoder will desire to identify the occurrence of VMX transitions. The packets of interests to a decoder are 
shown in Table 31-51. 
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Table 31-51. Packets on VMX Transitions (System-Wide Tracing) 


Event Packets Description 


VM exit FUP(GuestIP) The FUP indicates at which point in the quest flow the VM exit occurred. This is important, 
since VM exit can be an asynchronous event. The IP will match that written into the VMCS. 


PIP(HostCR3, NR=0) The PIP packet provides the new host CR3 value, as well as indication that the logical processor 
is entering VMX root operation. This allows the decoder to identify the change of executing 
context from guest to host and load the appropriate set of binaries to continue decode. 


TIP(HostIP) The TIP indicates the destination IP, the IP of the first instruction to be executed in VMX root 
operation. 

Note, this packet could be preceded by a MODE.Exec packet (Section 31.4.2.8). This is 
generated only in cases where CS.D or (CS.L & EFER.LMA) change during the transition. 


VM entry PIP(GuestCR3, NR=1) | The PIP packet provides the new guest CR3 value, as well as indication that the logical 
processor is entering VMX non-root operation. This allows the decoder to identify the change 
of executing context from host to quest and load the appropriate set of binaries to continue 
decode. 


TIP(GuestIP) The TIP indicates the destination IP, the IP of the first instruction to be executed in VMX non- 
root operation. This should match the RIP loaded from the VMCS. 


Note, this packet could be preceded by a MODE.Exec packet (Section 31.4.2.8). This is 
generated only in cases where CS.D or (CS.L & EFER.LMA) change during the transition. 


Since the VMX controls that suppress packet generation are cleared, a VMCS packet will be included in all PSB+ for 
this usage scenario. Additionally, VMPTRLD will generate such a packet. Thus the decoder can distinguish the 
execution context of different VMs. 


When the host VMM configures a system to collect trace packets in this scenario, it should emulate CPUID to report 
CPUID.(EAX=07H, ECX=0):EBX[bit 26] as 0 to guests, indicating to guests that Intel PT is not available. 


VMX TSC Manipulation 


The TSC packets generated while in VMX non-root operation will include any changes resulting from the use of a 
VMM’'s use of the TSC offsetting or TSC scaling VMX controls (see Chapter 24, “VMX Non-Root Operation”). In this 
system-wide usage model, the decoder may need to account for the effect of per-VM adjustments in the TSC 
packets generated in VMX non-root operation and the absence of TSC adjustments in TSC packets generated in 
VMX root operation. The VMM can supply this information to the decoder. 


31.5.2.2 Guest-Only Tracing 

A VMM can configure trace-packet generation while in VMX non-root operation for guests executing normally. This 
is accomplished by utilizing VMCS controls to manipulate the guest IA32_RTIT_CTL value on VMxX transitions. For 
older processors that do not support these VMCS controls, a VMM can use the VMX MSR-load areas on VM exits 
(see Section 23.7.2, “VM-Exit Controls for MSRs”) and VM entries (see Section 23.8.2, “VM-Entry Controls for 
MSRs”) to limit trace-packet generation to the guest environment. 

For this usage, VM-entry is programmed to enable trace packet generation, while VM-exit is programmed to clear 
IA32_RTIT_CTL.TraceEn so as to disable trace-packet generation in the host. Further, if it is preferred that the 
guest packet stream contain no indication that execution was in VMX non-root operation, the VMM should set to 
1 all the VMX controls enumerated in Table 31-50. 


31.5.2.3. Emulation of Intel PT Traced State 


If a VMM emulates an element of processor state by taking a VM exit on reads and/or writes to that piece of state, 
and the state element impacts Intel PT packet generation or values, it may be incumbent upon the VMM to insert 
or modify the output trace data. 


If a VM exit is taken on a guest write to CR3 (including “MOV CR3” as well as task switches), the PIP packet 
normally generated on the CR3 write will be missing. 
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To avoid decoder confusion when the guest trace is decoded, the VMM should emulate the missing PIP by writing it 
into the guest output buffer. If the guest CR3 value is manipulated, the VMM may also need to manipulate the 
IA32_RTIT_CR3_MATCH value, in order to ensure the trace behavior matches the guest's expectation. 


Similarly, if a VMM emulates the TSC value by taking a VM exit on RDTSC, the TSC packets generated in the trace 
may mismatch the TSC values returned by the VMM on RDTSC. To ensure that the trace can be properly aligned 
with software logs based on RDTSC, the VMM should either make corresponding modifications to the TSC packet 
values in the guest trace, or use mechanisms such as TSC offsetting or TSC scaling in place of exiting. 


31.5.2.4 TSC Scaling 


When TSC scaling is enabled for a guest using Intel PT, the VMM should ensure that the value of Maximum Non- 
Turbo Ratio[15:8] in MSR_PLATFORM_INFO (MSR OCEH) and the TSC/"core crystal clock” ratio (EBX/EAX) in CPUID 
leaf 15H are set in a manner consistent with the resulting TSC rate that will be visible to the VM. This will allow the 
decoder to properly apply TSC packets, MTC packets (based on the core crystal clock or ART, whose frequency is 
indicated by CPUID leaf 15H), and CBR packets (which indicate the ratio of the processor frequency to the Max 
Non-Turbo frequency). Absent this, or separate indication of the scaling factor, the decoder will be unable to prop- 
erly track time in the trace. See Section 31.8.3 for details on tracking time within an Intel PT trace. 


31.5.2.5 Failed VM Entry 


The packets generated by a failed VM entry depend both on the VMCS configuration, as well as on the type of 
failure. The results to expect are summarized in the table below. Note that packets in italics may or may not be 
generated, depending on implementation choice, and the point of failure. 


Table 31-52. Packets on a Failed VM Entry 


Usage Model | Entry Configuration Early Failure (fall Late Failure (VM-exit like) 
through to next IP) 
System-Wide | No use of “Load TIP (NextIP) PIP(Guest CR3, NR=1), TraceEn O->1 Packets (See Section 
IA32_RTIT_CTL” entry 31.2.7.3), PIP(HostCR3, NR=0), TIP(HostIP) 


control or VM-entry 
MSR-load area 


VMM Only “Load IA32_RTIT_CTL” | TIP (NextIP) TraceEn O->1 Packets (See Section 31.2.7.3), TIP(HostIP) 
entry control or VM- 
entry MSR-load area 
used to clear TraceEn 


VM Only “Load IA32_RTIT_CTL” | None None 
entry control or VM- 
entry MSR-load area 
used to set TraceEn 


31.5.2.6 VMX Abort 


VMX abort conditions take the processor into a shutdown state. On a VM exit that leads to VMX abort, some packets 
(FUP, PIP) may be generated, but any expected TIP, TIP.PGE, or TIP,PGD may be dropped. 


31.6 TRACING AND SMM TRANSFER MONITOR (STM) 


The SMM-transfer monitor (STM) is a VMM that operates inside SMM while in VMX root operation. An STM operates 
in conjunction with an executive monitor. The latter operates outside SMM and in VMX root operation. Transitions 
from the executive monitor or its VMs to the STM are called SMM VM exits. The STM returns from SMM via a 

VM entry to the VM in VMX non-root operation or the executive monitor in VMX root operation. 


Intel PT supports tracing in an STM similar to tracing support for VMX operation as described above in Section 31.5. 
As a result, on a SMM VM exit resulting from #SMI, TraceEn is neither saved nor cleared by default. Software can 
save the state of the trace configuration MSRs and clear TraceEn using the MSR load/save lists. 
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31.7. PACKET GENERATION SCENARIOS 


Table 31-53 and Table 31-55 illustrate the packets generated in various scenarios. In the heading row, PacketEn is 
abbreviated as PktEn, ContextEn as CntxEn. Note that this assumes that TraceEn=1 in IA32_RTIT_CTL, while Trig- 
gerEn=1 and Error=0 in IA32_RTIT_STATUS, unless otherwise specified. Entries that do not matter in packet 
generation are marked “D.C.” Packets followed by a “?” imply that these packets depend on additional factors, 
which are listed in the “Other Dependencies” column. 


There are additional scenarios, not covered below, where PSB+ packets (Section 31.3.7) may be generated. These 
include periodic PSB+ as well as use of IA32_RTIT_CTL.InjectPsbPmiOnEnable[56]=1 to preserve PSBs. 


The following acronyms are used in the packet examples below: 

° CLIP - Current LIP 

¢ NLIP - Next Sequential LIP 

¢ BLIP - Branch Target LIP 

In Table 31-53, PktEn is evaluated based on TriggerEn & ContextEn & FilterEn & BranchEn. 


Table 31-53. Packet Generation under Different Enable Conditions 


Case Operation PktEn | PktEn | Cntx€n Other Dependencies Packets Output 
Before | After | After 

la Normal non-jump operation 0 0 D.C. None 

1b Normal non-jump operation 1 1 1 None 

2a WRMSR/XRSTORS/RSM that changes | 0 0 D.C. *TSC if TSCEn=1; TSC?, TMA?, CBR 
TraceEn 0 -> 1, with PacketByteCnt >O *TMA if TSCEn=MTCEn=1 

2b WRMSR/XRSTORS/RSM that changes | 0 0 D.C. *TSC if TSCEn=1; PSB, PSBEND (see Sec- 
TraceEn 0 -> 1, with PacketByteCnt =0 *TMA if TSCEn=MTCEn=1 tion 31.4.2.17) 

2d WRMSR/XRSTORS/RSM that changes | 0 1 1 TSC if TSCEn=1; TSC?, TMA?, CBR, 

TraceEn 0 -> 1, with PacketByteCnt >0 TMA if TSCEn=MTCEn=1 MODE.Exec, TIP.PGE(NLIP) 

ze WRMSR/XRSTORS/RSM that changes | 0 1 1 MODE.Exec, 

TraceEn 0 -> 1, with PacketByteCnt =0 TIP.PGE(NLIP), PSB, 
PSBEND (see Section 
31.4.2.8, 31.4.2.7, 
31.4.2.13,31.4.2.15, 
31.42.17) 

3a WRMSR that changes TraceEn 1 -> 0 0 0 D.C. None 
3b WRMSR that changes TraceEn 1 -> O 1 0 D.C. FUP(CLIP), TIP.PGD() 
5a MOV to CR3 0 0 0 None 
5b MOV to CR3 0 1 1 *PIP.NR=1 if not in root PIP(NewCR3, NR?), 
operation and the “conceal | MODE.Exec?, 
VMX from PT” VM-execu- TIP.PGE(NLIP) 
tion control is 0 
*MODE.Exec if the mode has 
changed since the last 
MODE.Exec, or if no 
MODE.Exec since last PSB 
5c MOV to CR3 1 0 0 TIP.PGD() 
5d MOV to CR3 1 1 1 *PIP.NR=1 if not in root PIP(NewCR3, NR?) 
operation and the “conceal 
VMX from PT” VM-execu- 
tion control is 0 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | CntxEn Other Dependencies Packets Output 
Before | After | After 
5e MOV to CR3 1 0 1 *PIP.NR=1 if not in root PIP(NewCR3, NR?), 
operation and the “conceal | TIPPGD(NLIP), TraceStop? 
VMX from PT” VM-execu- 
tion control is O 
*TraceStop if executed in a 
TraceStop region 
of MOV to CR3 0 0 1 TraceStop if executed in a PIP(NewCR3,NR?), Trace- 
TraceStop region Stop? 
6a Unconditional direct near branch 0 0 D.C. None 
6b Unconditional direct near branch 1 0 1 TraceStop if BLIP is ina TIP.PGD(BLIP), TraceStop? 
TraceStop region 
6c Unconditional direct near branch 0 1 1 MODE.Exec if the mode has | MODE.Exec?, 
changed since the last TIP.PGE(BLIP) 
MODE.Exec, or if no 
MODE.Exec since last PSB 
6d Unconditional direct near branch 1 1 1 None 
7a Conditional taken jump or compressed | O 0) D.C. None 
RET that does not fill up the internal 
TNT buffer 
7b Conditional taken jump or compressed | O 1 1 MODE.Exec if the mode has | MODE.Exec?, 
RET changed since the last TIP.PGE(BLIP) 
MODE.Exec, or if no 
MODE.Exec since last PSB 
7d Conditional taken jump or compressed | 1 1 1 TNT 
RET that fills up the internal TNT buf- 
fer 
7e Conditional taken jump or compressed | 1 0 1 TraceStop if BLIP is ina TIP.PGD(), TraceStop? 
RET, with empty TNT buffer TraceStop region 
7f Conditional taken jump or compressed | 1 0 1 TraceStop if BLIP is ina TNT, TIP.PGD(), TraceS- 
RET, with non-empty TNT buffer TraceStop region top? 
8a Conditional non-taken jump 0 0 D.C. None 
8d Conditional not-taken jump that fills up | 1 1 1 TNT 
the internal TNT buffer 
9a Near indirect jump (JMP, CALL, or 0 0 D.C. None 
uncompressed RET) 
9b Near indirect jump (JMP, CALL, or 0 1 1 MODE.Exec if the mode has | MODE.Exec?, 
uncompressed RET) changed since the last TIP.PGE(BLIP) 
MODE.Exec, or if no 
MODE.Exec since last PSB 
9c Near indirect jump (JMP, CALL, or 1 0) 1 TraceStop if BLIP is ina TIP.PGD(BLIP), TraceStop? 
uncompressed RET) TraceStop region 
Sd Near indirect jump (JMP, CALL, or 1 1 1 TIP(BLIP) 
uncompressed RET) 
10a_ | Far Branch (CALL/JMP/RET/SYS*/IRET) | 0 0 0 None 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case 


Operation 


PktEn 
Before 


PktEn 
After 


CntxEn 


After 


Other Dependencies 


Packets Output 


10b 


Far Branch (CALL/JMP/RET/SYS*/IRET) 


0 


1 


1 


*PIP if CR3 is updated (i.e, 
task switch), and OS=1; 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*MODE.Exec if the mode has 
changed since the last 
MODE.Exec, or if no 
MODE.Exec since last PSB 


PIP(new CR3, NR?), 
MODE.Exec?, 
TIP.PGE(BLIP) 


10c 


Far Branch (CALL/JMP/RET/SYS*/IRET) 


TIPPGD() 


10d 


10e 


Far Branch (CALL/JMP/RET/SYS*/IRET) 


Far Branch (CALL/JMP/RET/SYS*/IRET) 


*PIP if CR3 is updated (ie., 
task switch), and OS=1; 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


*PIP if CR3 is updated (ie., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 

* MODE.Exec if the opera- 
tion changes CS.L/D or 
IA32_EFER.LMA 


PIP(new CR3, NR?), 
TIP.PGD(BLIP), TraceStop? 


PIP(NewCR3, NR?)?, 
MODE.Exec?, TIP(BLIP) 


10f 


Far Branch (CALL/JMP/RET/SYS*/IRET) 


*PIP if CR3 is updated (i.e., 
task switch), and OS=1; 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


PIP(new CR3, NR?), Trace- 
Stop? 


11a 


HW Interrupt 


None 


11b 


HW Interrupt 


*PIP if CR3 is updated (i.e, 
task switch), and OS=1; 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 

* MODE.Exec if the mode 
has changed since the last 
MODE.Exec, or if no 
MODE.Exec since last PSB 


PIP(new CR3, NR?), 
MODE.Exec?, 
TIP.PGE(BLIP) 


HW Interrupt 


FUP(NLIP), TIP.PGD() 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case Operation 


PktEn 
Before 


PktEn 
After 


CntxEn 
After 


Other Dependencies 


Packets Output 


11d | HW Interrupt 


1 


0 


1 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


FUP(NLIP), PIP(NewCR3, 
NR?)?, TIPPGD(BLIP), 
TraceStop? 


11e | HW Interrupt 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 

* MODE.Exec if the opera- 
tion changes CS.L/D or 
IA32_EFER.LMA 


FUP(NLIP), PIP(NewCR3, 
NR?)?, MODE.Exec?, 
TIP(BLIP) 


11f | HW Interrupt 


12a | SW Interrupt 


*PIP if CR3 is updated (ie., 
task switch), and OS=1; 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


PIP(new CR3, NR?), Trace- 
Stop? 


None 


12b | SW Interrupt 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*MODE.Exec if the mode has 
changed since the last 
MODE.Exec, or if no 
MODE.Exec since last PSB 


PIP(NewCR3, NR?)?, 
MODE.Exec?, 
TIP.PGE(BLIP) 


12c | SW Interrupt 


FUP(CLIP), TIPPGD() 


12d | SW Interrupt 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


FUP(CLIP), PIP(NewCR3, 
NR?)?, TIPPGD(BLIP), 
TraceStop? 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case 


Operation 


PktEn 
Before 


PktEn 
After 


CntxEn 
After 


Other Dependencies 


Packets Output 


12e 


SW Interrupt 


1 


1 


1 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 

* MODE.Exec if the opera- 
tion changes CS.L/D or 
IA32_EFER.LMA 


FUP(CLIP), PIP(NewCR3, 
NR?)?, MODE.Exec?, 
TIP(BLIP) 


12f 


SW Interrupt 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


PIP(NewCR3, NR?)?, 
TraceStop? 


13a 


Exception/Fault 


None 


13b 


Exception/Fault 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*MODE.Exec if the mode has 
changed since the last 
MODE.Exec, or if no 
MODE.Exec since last PSB 


PIP(NewCR3, NR?)?, 
MODE.Exec?, 
TIP.PGE(BLIP) 


13c 
13d 


Exception/Fault 


Exception/Fault 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 


FUP(CLIP), TIPPGD() 


FUP(CLIP), PIP(NewCR3, 
NR?)?, TIPPGD(BLIP), 
TraceStop? 


Exception/Fault 


* PIP if CR3 is updated (i.e., 
task switch), and OS=1 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 

* MODE.Exec if the opera- 
tion changes CS.L/D or 
IA32_EFER.LMA 


FUP(CLIP), PIP(NewCR3, 
NR?)?, MODE.Exec?, 
TIP(BLIP) 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | CntxEn Other Dependencies Packets Output 
Before | After | After 
13f | Exception/Fault 0 0 1 * PIP if CR3 is updated (i.e., | PIP(NewCR3, NR?)?, 
task switch), and OS=1 TraceStop? 
*PIP.NR=1 if destination is 
not root operation and the 
“conceal VMX from PT” VM- 
execution control is 0; 
*TraceStop if BLIP is ina 
TraceStop region 
14a | SMI (TraceEn cleared) 0 0 D.C. None 
14b | SMI (TraceEn cleared) 1 0 FUP(SMRAM.LIP), 
TIP.PGD() 
14c | SMI (TraceEn cleared) 1 1 1 NA 
14f | SMI (TraceEn cleared) 1 0 1 NA 
15a _| RSM, TraceEn restored to 0 0 0 None 
15b_ | RSM, TraceEn restored to 1 0 0 D.C. See WRMSR cases for 
packets on enable 
15c_ | RSM, TraceEn restored to 1 0 1 1 See WRMSR cases for 
packets on enable. 
FUP/TIP.PGE IP is 
SMRAM.LIP 
15d_ | RSM (TraceEn=1, goes to shutdown) 1 1 1 None 
15e | RSM (TraceEn=1, goes to shutdown) 1 0 0 None 
15f | RSM (TraceEn=1, goes to shutdown) 1 0 1 None 
16a | VMexit 0 0 1 *PIP if OF=1 and the “con- ‘| PIP(HostCR3, NR=0)?, 
ceal VMX from PT” VM-exit | TraceStop? 
control is 0; 
*TraceStop if VMCSHLLIP is 
in a TraceStop region 
16b | VM exit, MSR list sets TraceEn=1 0 0 0 See WRMSR cases for 
packets on enable. FUP IP 
is VMCSh.LIP 
16c_ | VM exit, MSR list sets TraceEn=1 0 1 1 See WRMSR cases for 
packets on enable. 
FUP/TIP.PGE IP is 
VMCSh.LIP 
16e | VMexit 0 1 1 *PIP if OF=1 and the “con- ‘| PIP(HostCR3, NR=0)?, 
ceal VMX from PT” VM-exit | MODE.Exec?, 
control is 0; TIP.PGE(VMCSh.LIP) 
*MODE.Exec if the value is 
different, since last TIRPGD 
16f | VM exit, MSR list clears TraceEn=0 1 0 0 *PIP if OF=1 and the “con- | FUP(VMCSg.LIP), 


ceal VMX from PT” VM-exit 
control is 0; 


PIP(HostCR3, NR=0)?, 
TIP.PGD 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | Cntx€n Other Dependencies Packets Output 
Before | After | After 
16g | VMexit 1 0 1 *PIP if OF=1 and the “con- | FUP(VMCSg.LIP), 
ceal VMX from PT” VM-exit | PIP(HostCR3, NR=O0)?, 
control is 0; TIP.PGD(VMCSh.LIP), 
*TraceStop if VMCSh.LIP is | TraceStop? 
in a TraceStop region 
16h | VMexit 1 1 1 *PIP if OF=1 and the “con- | FUP(VMCSg.LIP), 
ceal VMX from PT” VM-exit | PIP(HostCR3, NR=0)?, 
control is 0; MODE.Exec, 
*MODE.Exec if the value is — | TIP(VMCSh.LIP) 
different, since last TIRPGD 
16i | VM exit 0 0 0 None 
16) | VMexit, ContextE€N 1->0 1 0 0 FUP(VMCSg.LIP), TIP.PGD 
17a | VMentry 0 0 0 None 
17b | VM entry 0 0 1 *PIP if OF=1 and the “con- ‘| PIP(GuestCR3, NR=1)?, 
ceal VMX from PT” VM- TraceStop? 
entry control is 0; 
*TraceStop if VMCSg.LIP is 
in a TraceStop region 
17c | VMentry, MSR load list sets 0 0 1 See WRMSR cases for 
TraceEn=1 packets on enable. FUP IP 
is VMCSg.LIP 
17d | VMentry, MSR load list sets 0 1 1 See WRMSR cases for 
TraceEn=1 packets on enable. 
FUP/TIP.PGE IP is 
VMCSg.LIP 
17f | VMentry, Filter—EN O->1 0 1 1 *PIP if OF=1 andthe “con- ‘| PIP(GuestCR3, NR=1)?, 
ceal VMX from PT” VM- MODE.Exec?, 
entry control is 0; TIP.PGE(VMCSg.LIP) 
*MODE.Exec if the value is 
different, since last TIRPGD 
17g | VMentry, MSR list clears TraceEn=0 1 0 0 *PIP if OF=1 and the “con- ‘| PIP(GuestCR3, NR=1)?, 
ceal VMX from PT” VM- TIP.PGD 
entry control is 0; 
17h | VM entry 1 0 1 *PIP if OF=1 and the “con- ‘| PIP(GuestCR3, NR=1)?, 
ceal VMX from PT” VM- TIP.PGD(VMCSg.LIP), 
entry control is 0; TraceStop? 
*TraceStop if VMCSg.LIP is 
in a TraceStop region 
171 | VMentry 1 1 1 *PIP if OF=1 and the “con- ‘| PIP(GuestCR3, NR=1)?, 
ceal VMX from PT” VM- MODE.Exec, 
entry control is 0; TIP(VMCSg.LIP) 
*MODE.Exec if the value is 
different, since last TIRPGD 
17j | VMentry, ContextEN O->1 0 1 1 *MODE.Exec if the value is MODE.Exec, 
different, since last TIRPGD | TIP,PGE(VMCSg.LIP) 
20a_| EENTER/ERESUME to non-debug 0 0 0 None 


enclave 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | CntxEn Other Dependencies Packets Output 
Before | After | After 
20c_ | EENTER/ERESUME to non-debug 1 0 0 FUP(CLIP), TIP.PGD() 
enclave 
21a_ | EEXIT from non-debug enclave 0 D.C. None 
21b | EEXIT from non-debug enclave 1 1 *MODE.Exec if the value is | MODE.Exec?, 
different, since last TIPPGD | TIPPGE(BLIP) 
22a_ | AEX/EEE from non-debug enclave 0 D.C. None 
22b | AEX/EEE from non-debug enclave 1 1 *MODE.Exec if the value is | MODE.Exec?, 
different, since last TIRPGD | TIP,PGE(AEP.LIP) 
23a _ | EENTER/ERESUME to debug enclave 0 D.C. None 
23b | EENTER/ERESUME to debug enclave 1 1 *MODE.Exec if the value is MODE.Exec?, 
different, since last TIRPGD | TIPPGE(BLIP) 
23c_ | EENTER/ERESUME to debug enclave 1 0 0 FUP(CLIP), TIP.PGD() 
23d | EENTER/ERESUME to debug enclave 0 0 1 *TraceStop if BLIP is ina FUP(CLIP), TIP.PGD(BLIP), 
TraceStop region TraceStop? 
23e | EENTER/ERESUME to debug enclave 1 1 1 FUP(CLIP), TIP(BLIP) 
24b | EEXIT from debug enclave 0 1 1 *MODE.Exec if the value is MODE.Exec?, 
different, since last TIRPGD | TIPPGE(BLIP) 
24d | EEXIT from debug enclave 1 0 1 *TraceStop if BLIP is in a FUP(CLIP), TIP.PGD(BLIP), 
TraceStop region TraceStop? 
24e | EEXIT from debug enclave 1 1 1 FUP(CLIP), TIP(BLIP) 
24f | EEXIT from debug enclave 0 0 D.C. None 
25a_ | AEX/EEE from debug enclave 0 0 D.C. None 
25b | AEX/EEE from debug enclave 0 1 1 *MODE.Exec if the value is MODE.Exec?, 
different, since last TIRPGD | TIP,PGE(AEP.LIP) 
25d_ | AEX/EEE from debug enclave 1 0 1 *For AEX, FUP IP could be FUP(CLIP), 
NLIP, for trap-like events TIP.PGD(AEP.LIP) 
25e | AEX/EEE from debug enclave 1 1 1 *MODE.Exec if the value is | FUP(CLIP), MODE.Exec?, 
different, since last TIPPGD | TIP(AEP.LIP) 
*For AEX, FUP IP could be 
NLIP, for trap-like events 
26a _ | XBEGIN/XACQUIRE 0 0 D.C. None 
26d | XBEGIN/XACQUIRE that does not set | 1 1 1 None 
InTX 
26e | XBEGIN/XACQUIRE that sets InTX 1 1 1 MODE.TSX(INTX=1, 
TXAbort=0), FUP(CLIP) 
27a _ | XEND/XRELEASE 0 0 D.C. None 
27d | XEND/XRELEASE that does not clear 1 1 1 None 
InTX 
27e | XEND/XRELEASE that clears InTX 1 1 1 MODE.TSX(InTX=0, 
TXAbort=0), FUP(CLIP) 
28a | XABORT(Async XAbort, or other) 0 0 0 None 
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Table 31-53. Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | Cntx€n Other Dependencies Packets Output 
Before | After | After 
28b | XABORT(Async XAbort, or other) 0 1 1 MODE.TSX(InTX=0, 
TXAbort=1), 
TIP.PGE(BLIP) 
28c_ | XABORT(Async XAbort, or other) 1 0 1 *TraceStop if BLIP is ina MODE.TSX(InTX=0, 
TraceStop region TXAbort=1), TIP-PGD 
(BLIP), TraceStop? 
28d | XABORT(Async XAbort, or other) 1 1 1 MODE.TSX(InTX=0, 
TXAbort=1), FUP(CLIP), 
TIP(BLIP) 
28e | XABORT(Async XAbort, or other) 0 0 1 *TraceStop if BLIP is ina MODE.TSX(InTX=0, 
TraceStop region TXAbort=1), TraceStop? 
30a _| INIT (BSP) 0 0 None 
30b_ | INIT (BSP) 0 1 *TraceStop if RESET.LIP is in | PIP(O), TraceStop? 
a TraceStop region 
30c_ | INIT (BSP) 0 1 1 * MODE.Exec if the value is | MODE.Exec?, PIP(0), 
different, since last TIRPGD | TIP.PGE(ResetLIP) 
30d _ | INIT (BSP) 1 0 0 FUP(NLIP), TIP.PGD() 
30e | INIT (BSP) 1 0 1 * PIP if OS=1 FUP(NLIP), PIP(0), 
*TraceStop if RESET.LIP is in | TIPPGD, TraceStop? 
a TraceStop region 
30f | INIT (BSP) 1 1 1 * MODE.Exec if the mode FUP(NLIP), PIP(O)?, 
has changed since the last | MODE.Exec?, 
MODE.Exec, or if no TIP(ResetLIP) 
MODE.Exec since last PSB 
* PIP if OS=1 
31a _| INIT (AP, goes to wait-for-SIPI) 0 D.C. D.C. None 
31b_ | INIT (AP, goes to wait-for-SIPl) 1 D.C. D.C. * PIP if OS=1 FUP(NLIP), PIP(O) 
32a _ | SIPI 0 0 0 None 
32c | SIPI 0 1 1 * MODE.Exec if the mode MODE.Exec?, TIP.PGE(SIPI- 
has changed since the last | LIP) 
MODE.Exec, or if no 
MODE.Exec since last PSB 
32d | SIPI 1 0 0 TIP.PGD 
32e | SIPI 1 0 1 *TraceStop if SIPI LIP is ina | TIP,PGD(SIPILIP); TraceS- 
TraceStop region top? 
32f | SIPI 1 1 1 * MODE.Exec if the mode MODE.Exec?, TIP(SIPILIP) 
has changed since the last 
MODE.Exec, or if no 
MODE.Exec since last PSB 
33a _ | MWAIT (to CO) D.C. D.C. D.C. None 
33b | MWAIT (to higher-numbered C-State, | D.C. D.C. D.C. *TSC if TSCEn=1 TSC?, TMA?, CBR 


packet sent on wake) 


*TMA if TSCEn=MTCEn=1 


In Table 31-54, PktEn is evaluated based on (TriggerEn & ContextEn & PwrEvtEn). 
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Table 31-54. Pwr€vtEn and PTWEn Packet Generation under Different Enable Conditions 


Case Operation PktEn | PktEn | CntxEn Other Dependencies Packets Output 
Before | After | After 
16.1 MWAIT or 1/0 redir to MWAIT, gets | D.C. D.C. D.C. None 
#UD or #GP fault 
16.2 MWAIT or I/O redir to MWAIT, D.C. D.C. D.C. See VM exit examples 
VM exits (16[a-z] in Table 31-53) 
for BranchEn packets. 
16.3 MWAIT or 1/0 redir to MWAIT, D.C. D.C. D.C. None 
requests CO, or monitor not armed, 
or VMX virtual-interrupt delivery 
16.4a | MWAIT(X) or 1/0 redir to MWAIT, D.C. 0 0 PWRE(Cx), EXSTOP 
goes to C-state Y (Y>0) 
16.4b | MWAIT(X) or 1/0 redir to MWAIT, D.C. D.C. 1 MWAIT(Cy), PWRE(Cx), 
goes to C-state Y (Y>0) EXSTOP(IP), FUP(CLIP) 
16.5a | MWAIT(X) or 1/0 redir to MWAIT, D.C. 0 0 * TSC if TSCEn=1 PWRE(Cx), EXSTOP, TSC?, 
Pending event after resolving to go * TMA if TSCEn=MTCEn=1 TMA?, CBR, PWRX(LCC, 
to C-state Y (Y>0) DCC, 0) 
16.5b | MWAIT(X) or 1/0 redir to MWAIT, D.C. D.C. 1 * TSC if TSCEn=1 PWRE(Cx), EXSTOP(IP), 
Pending event after resolving to go * TMA if TSCEn=MTCEn=1 FUP(CLIP), TSC?, TMA?, 
to C-state Y (Y>0) CBR, PWRX(LCC, DCC, 0) 
16.6a | MWAIT(5) or I/O redir to MWAIT, D.C. 0 0 PWRE(C1), EXSTOP 
other thread(s) in core in CO/C1 
16.6b | MWAIT(5) or I/0 redir to MWAIT, D.C. D.C. 1 MWAIT(5), PWRE(C1), 
other thread(s) in core in CO/C1 EXSTOP(IP), FUP(CLIP) 
16.9a | HLT, Triple-fault shutdown, #MC D.C. 0 0 PWRE(C1), EXSTOP 
with CR4.MCE=0, RSM to Cx (x>0) 
16.9b | HLT, Triple-fault shutdown, #MC D.C. D.C. PWRE(C1), EXSTOP(IP), 
with CR4.MCE=1, RSM to Cx (x>0) FUP(CLIP) 
16.10a | VMX abort D.C. 0 0 See “VMX Abort" (cases 
16* and 18* in Table 31- 
53) for BranchEn packets 
that precede 
PWRE(C1), EXSTOP 
16.10b | VMX abort D.C. D.C. 1 See “VMX Abort" (cases 
16* and 18% in Table 31- 
53) for BranchEn packets 
that precede 
PWRE(C1), EXSTOP(IP), 
FUP(CLIP) 
16.11a | RSM to Shutdown D.C. 0 0 See “RSM to Shutdown” 


(cases 15[def] in Table 
31-53) for BranchEn 
packets that precede 


PWRE(C1), EXSTOP 
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Table 31-54. PwrE€vtEn and PTWEn Packet Generation under Different Enable Conditions (Contd.) 


Case 


Operation 


PktEn 
Before 


PktEn 
After 


CntxEn 
After 


Other Dependencies 


Packets Output 


16.11b 


RSM to Shutdown 


D.C. 


D.C. 


1 


See “RSM to Shutdown” 
(cases 15[def] in Table 
31-53) for BranchEn 
packets that precede 


PWRE(C1), EXSTOP(IP), 
FUP(CLIP) 


16.12a 


INIT (BSP) 


D.C. 


See “INIT (BSP)” (cases 
30[a-z] in Table 31-53) 
for BranchEn packets that 
precede 


PWRE(C1), EXSTOP 


16.12b 


INIT (BSP) 


D.C. 


D.C. 


See “INIT (BSP)” (cases 
30[a-z] in Table 31-53) 
for BranchEn packets that 
precede 


PWRE(C1), EXSTOP(IP), 
FUP(NLIP) 


16.13a 


INIT (AP, goes to Wait-for-SIPI) 


D.C. 


See “INIT (AP, goes to 
Wait-for-SIPl)” (cases 
31[a-z] in Table 31-53) 
for BranchEn packets that 
precede 


PWRE(C1), EXSTOP 


16.13b 


INIT (AP, goes to Wait-for-SIPI) 


D.C. 


D.C. 


See “INIT (AP, goes to 
Wait-for-SIPl)” (cases 
31[a-z] in Table 31-53) 
for BranchEn packets that 
precede 


PWRE(C1), EXSTOP(IP), 
FUP(NLIP) 


16.14a 


Hardware Duty Cycling (HDC) 


D.C. 


* TSC if TSCEn=1 
* TMA if TSCEn=MTCEn=1 


PWRE(HW, C6), EXSTOP, 
TSC?, TMA?, CBR, 
PWRX(CC6, CC6, Ox8) 


16.14b 


Hardware Duty Cycling (HDC) 


D.C. 


D.C. 


* TSC if TSCEn=1 
* TMA if TSCEn=MTCEn=1 


PWRE(HW, C6), EXS- 
TOP(IP), FUP(NLIP), TSC?, 
TMA?, CBR, PWRX(CCE, 
CC6, 0x8) 


16.15a 


VM entry to HLT or Shutdown 


D.C. 


See “VM entry” (cases 
17[a-z] in Table 31-53) 
for BranchEn packets that 
precede 


PWRE(C1), EXSTOP 
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Table 31-54. PwrE€vtEn and PTWEn Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | Cntx€n Other Dependencies Packets Output 
Before | After | After 
16.15b | VM entry to HLT or Shutdown D.C. D.C. 1 See “VM entry” (cases 
17[a-z] in Table 31-53) 
for BranchEn packets that 
precede 
PWRE(C1), EXSTOP(IP), 
FUP(CLIP) 
16.16a | EIST in CO, S1/TM1/TM2, or STP- D.C. 0 0 * TSC if TSCEn=1 EXSTOP, TSC?, TMA?, CBR 
CLK# * TMA if TSCEn=MTCEn=1 
16.16b | EIST in CO, S1/TM1/TM2, or STP- D.C. D.C. 1 * TSC if TSCEn=1 EXSTOP(IP), FUP(NLIP), 
CLK# * TMA if TSCEn=MTCEn=1 TSC?, TMA?, CBR 
16.17 | EIST in Cx (x>0) D.C. D.C. D.C. None 
16.18 | INTR during Cx (x>0) D.C. D.C. D.C. * TSC if TSCEn=1 TSC?, TMA?, CBR, 
* TMA if TSCEn=MTCEn=1 PWRX(LCC, DCC, 0x1) 
See “HW Interrupt” (cases 
11[a-z] in Table 31-53) 
for BranchEn packets that 
follow. 
16.18 | SMI during Cx (x>0) D.C. D.C. D.C. * TSC if TSCEn=1 TSC?, TMA?, CBR, 
* TMA if TSCEn=MTCEn=1 PWRX(LCC, DCC, 0) 
See “HW Interrupt” (cases 
14[a-z] in Table 31-53) 
for BranchEn packets that 
follow. 
16.19 | NMI during Cx (x>0) D.C. D.C. D.C. * TSC if TSCEn=1 TSC?, TMA?, CBR, 
* TMA if TSCEn=MTCEn=1 PWRX(LCC, DCC, 0) 
See “HW Interrupt” (cases 
11[a-z] in Table 31-53) 
for BranchEn packets that 
follow. 
16.20 | Store to monitored address during | D.C. D.C. D.C. * TSC if TSCEn=1 TSC?, TMA?, CBR, 
Cx (x>0) * TMA if TSCEn=MTCEn=1 PWRX(LCC, DCC, 0x4) 
16.22 | #MC, IERR, TSC deadline timer D.C. D.C. D.C. * TSC if TSCEn=1 TSC?, TMA?, CBR, 
expiration, or APIC counter under- * TMA if TSCEn=MTCEn=1 PWRX(LCC, DCC, 0) 
flow during Cx (x>0) 


In Table 31-55, PktEn is evaluated based on (TriggerEn & ContextEn & FilterEn & PTWEn). 


Table 31-55. Pwr€vtEn and PTWEn Packet Generation under Different Enable Conditions 


Case Operation PktEn | PktEn | Cntx€n Other Dependencies Packets Output 
Before | After | After 
16.24a | PTWRITE rm32/64, no fault D.C. D.C. D.C. None 
16.24b | PTWRITE rm32/64, no fault D.C. 0 0 None 
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Table 31-55. PwrEvtEn and PTWEn Packet Generation under Different Enable Conditions (Contd.) 


Case Operation PktEn | PktEn | Cntx€n Other Dependencies Packets Output 
Before | After | After 

16.24d | PTWRITE rm32, no fault D.C. 1 1 * FUP, IP=1 if FUPonPTW=1 | PTW(IP=1?, 4B, 
rm32_value), FUP(CLIP)? 

16.24e | PTWRITE rm64, no fault D.C. 1 1 * FUP, IP=1 if FUPonPTW=1 | PTW(IP=1?, 8B, 
rm64_value), FUP(CLIP)? 

16.25a | PTWRITE mem32/64, fault D.C. D.C. D.C. See “Exception/fault” 
(cases 13[a-z] in Table 
31-53) for BranchEn 
packets. 


31.8 SOFTWARE CONSIDERATIONS 


31.8.1 Tracing SMM Code 


Nothing prevents an SMM handler from configuring and enabling packet generation for its own use. As described in 
Section Section 31.2.8.3, SMI will always clear TraceEn, so the SMM handler would have to set TraceEn in order to 
enable tracing. There are some unique aspects and guidelines involved with tracing SMM code, which follow: 


1. SMM should save away the existing values of any configuration MSRs that SMM intends to modify for tracing. 
This will allow the non-SMM tracing context to be restored before RSM. 


2. Itis recommended that SMM wait until it sets CSbase to 0 before enabling packet generation, to avoid possible 
LIP vs RIP confusion. 


3. Packet output cannot be directed to SMRR memory, even while tracing in SMM. 


Before performing RSM, SMM should take care to restore modified configuration MSRs to the values they had 
immediately after #SMI. This involves first disabling packet generation by clearing TraceEn, then restoring any 
other configuration MSRs that were modified. 


5. RSM 


— Software must ensure that TraceEn=0 at the time of RSM. Tracing RSM is not a supported usage model, and 
the packets generated by RSM are undefined. 


— For processors on which Intel PT and LBR use are mutually exclusive (see Section 31.3.1.2), any RSM 
during which TraceEn is restored to 1 will suspend any LBR or BTS logging. 


31.8.2 Cooperative Transition of Multiple Trace Collection Agents 


A third-party trace-collection tool should take into consideration the fact that it may be deployed on a processor 
that supports Intel PT but may run under any operating system. 


In such a deployment scenario, Intel recommends that tool agents follow similar principles of cooperative transition 
of single-use hardware resources, similar to how performance monitoring tools handle performance monitoring 
hardware: 


¢ Respect the “in-use” ownership of an agent who already configured the trace configuration MSRs, see architec- 
tural MSRs with the prefix “IA32_RTIT_” in Chapter 2, “Model-Specific Registers (MSRs)” in the Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 4, where “in-use” can be determined by reading the 
“enable bits” in the configuration MSRs. 


° ~Relinquish ownership of the trace configuration MSRs by clearing the “enabled bits” of those configuration 
MSRs. 
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31.8.3. Tracking Time 


This section describes the relationships of several clock counters whose update frequencies reside in different 
domains that feed into the timing packets. To track time, the decoder also needs to know the regularity or irregu- 
larity of the occurrences of various timing packets that store those clock counters. 


Intel PT provides time information for three different but related domains: 
¢ Processor timestamp counter 


This counter increments at the max non-turbo or P1 ical uad and its value is returned on a RDTSC. Its 
frequency is fixed. The TSC packet holds the lower 7 bytes of the timestamp counter value. The TSC packet 
occurs occasionally and are much less frequent than the frequency of the time stamp counter. The timestamp 
counter will continue to increment when the processor is in deep C-States, with the exception of processors 
reporting CPUID.80000007H:EDX.InvariantTSC[bit 8] =0. 


* Core crystal clock 


The ratio of the core crystal clock to timestamp counter frequency is known as P, and can be calculated as 
CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0]. The frequency of the core crystal clock is fixed and lower than 
that of the timestamp counter. The periodic MTC packet is generated based on software-selected multiples of 
the crystal clock frequency. The MTC packet is expected to occur more frequently than the TSC packet. 


¢ Processor core clock 


The processor core clock frequency can vary due to P-state and thermal conditions. The CYC packet provides 
elapsed time as measured in processor core clock cycles relative to the last CYC packet. 


A decoder can use all or some combination of these packets to track time at different resolutions throughout the 
trace packets. 


31.8.3.1 Time Domain Relationships 
The three domains are related by the following formula: 
TimeStampValue = (CoreCrystalClockValue * P) + AdjustedProcessorCycles + Software Offset; 


The CoreCrystalClockValue, also known as the Always Running Timer (ART) value, can provide the coarse-grained 
component of the TSC value. P, or the TSC/ART ratio, can be derived from CPUID leaf 15H, as described in Section 
31.8.3. 


The AdjustedProcessorCycles component provides the fine-grained distance from the rising edge of the last core 

crystal clock. Specifically, it is a cycle count in the same frequency as the timestamp counter from the last crystal 
clock rising edge. The value is adjusted based on the ratio of the processor core clock frequency to the Maximum 
Non-Turbo (or P1) frequency. 


The Software_Offsets component includes software offsets that are factored into the timestamp value, such as 
IA32_TSC_ADJUST. 


31.8.3.2 Estimating TSC within Intel PT 


For many usages, it may be useful to have an estimated timestamp value for all points in the trace. The formula 
provided in Section 31.8.3.1 above provides the framework for how such an estimate can be calculated from the 
various timing packets present in the trace. 


The TSC packet provides the precise timestamp value at the time it is generated; however, TSC packets are infre- 
quent, and estimates of the current timestamp value based purely on TSC packets are likely to be very inaccurate 
for this reason. In order to get more precise timing information between TSC packets, CYC packets and/or MTC 
packets should be enabled. 


MTC packets provide incremental updates of the CoreCrystalClockValue. On processors that support CPUID leaf 
15H, the frequency of the timestamp counter and the core crystal clock is fixed, thus MTC packets provide a means 
to update the running timestamp estimate. Between two MTC packets A and B, the number of crystal clock cycles 
passed is calculated from the 8-bit payloads of respective MTC packets: 


(CTCp - CTCa), where CTC; = MTC[15:8] << IA32_RTIT_CTL.MTCFreq and i = A, B. 


The time from a TSC packet to the subsequent MTC packet can be calculated using the TMA packet that follows the 
TSC packet. The TMA packet provides both the crystal clock value (lower 16 bits, in the CTC field) and the Adjust- 
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edProcessorCycles value (in the FastCounter field) that can be used in the calculation of the corresponding core 
crystal clock value of the TSC packet. 


When the next MTC after a pair of TSC/TMA is seen, the number of crystal clocks passed since the TSC packet can 
be calculated by subtracting the TMA.CTC value from the time indicated by the MTCy.,; packet by 


CTCpettal 15:0] = (CTCyextl15:0] - TMA.CTC[15:0]), where CTCyext = MTCpayload << IA32_RTIT_CTL.MTCFreq. 


The TMA.FastCounter field provides the number of AdjustedProcessorCycles since the last crystal clock rising edge, 
from which it can be determined the percentage of the next crystal clock cycle that had passed at the time of the 
TSC packet. 


CYC packets can provide further precision of an estimated timestamp value to many non-timing packets, by 
providing an indication of the time passed between other timing packets (MTCs or TSCs). 


When enabled, CYC packets are sent preceding each CYC-eligible packet, and provide the number of processor 
core clock cycles that have passed since the last CYC packet. Thus between MTCs and TSCs, the accumulated CYC 
values can be used to estimate the AdjustedProcessorCycles component of the timestamp value. The accumulated 
CPU cycles will have to be adjusted to account for the difference in frequency between the processor core clock and 
the Pi frequency. The necessary adjustment can be estimated using the core:bus ratio value given in the CBR 
packet, by multiplying the accumulated cycle count value by P1/CBRpayload: 


Note that stand-alone TSC packets (that is, TSC packets that are not a part of a PSB+) are typically generated only 
when generation of other timing packets (MTCs and CYCs) has ceased for a period of time. Example scenarios 
include when Intel PT is re-enabled, or on wake after a sleep state. Thus any calculated estimate of the timestamp 
value leading up to a TSC packet will likely result in a discrepancy, which the TSC packet serves to correct. 


A greater level of precision may be achieved by calculating the CPU clock frequency, see Section 31.8.3.4 below for 
a method to do so using Intel PT packets. 


CYCs can be used to estimate time between TSCs even without MTCs, though this will likely result in a reduction in 
estimated TSC precision. 


31.8.3.3. VMX TSC Manipulation 


When software executes in non-Root operation, additional offset and scaling factors may be applied to the TSC 
value. These are optional, but may be enabled via VMCS controls on a per-VM basis. See Chapter 24, “VMX Non- 
Root Operation” for details on VMX TSC offsetting and TSC scaling. 


Like the value returned by RDTSC, TSC packets will include these adjustments, but other timing packets (such as 
MTC, CYC, and CBR) are not impacted. In order to use the algorithm above to estimate the TSC value when TSC 
scaling is in use, it will be necessary for software to account for the scaling factor. See Section 31.5.2.4 for details. 


31.8.3.4 Calculating Frequency with Intel PT 


Because Intel PT can provide both wall-clock time and processor clock cycle time, it can be used to measure the 
processor core clock frequency. Either TSC or MTC packets can be used to track the wall-clock time. By using CYC 
packets to count the number of processor core cycles that pass in between a pair of wall-clock time packets, the 
ratio between processor core clock frequency and TSC frequency can be derived. If the Pi frequency is known, it 
can be applied to determine the CPU frequency. See Section 31.8.3.1 above for details on the relationship between 
TSC, MTC, and CYC. 
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CHAPTER 32 
INTRODUCTION TO INTEL® SOFTWARE GUARD EXTENSIONS 


32.1 OVERVIEW 


Intel® Software Guard Extensions (Intel® SGX) is a set of instructions and mechanisms for memory accesses 
added to Intel® Architecture processors. Intel SGX can encompass two collections of instruction extensions, 
referred to as SGX1 and SGX2, see Table 32-1 and Table 32-2. The SGX1 extensions allow an application to instan- 
tiate a protected container, referred to as an enclave. The enclave is a trusted area of memory, where critical 
aspects of the application functionality have hardware-enhanced confidentiality and integrity protections. New 
access controls to restrict access to software not resident in the enclave are also introduced. The SGX2 extensions 
allow additional flexibility in runtime management of enclave resources and thread execution within an enclave. 


Chapter 33 covers main concepts, objects and data structure formats that interact within the Intel SGX architec- 
ture. Chapter 34 covers operational aspects ranging from preparing an enclave, transferring control to enclave 
code, and programming considerations for the enclave code and system software providing support for enclave 
execution. Chapter 35 describes the behavior of Asynchronous Enclave Exit (AEX) caused by events while 
executing enclave code. Chapter 36 covers the syntax and operational details of the instruction and associated leaf 
functions available in Intel SGX. Chapter 37 describes interaction of various aspects of IA32 and Intel® 64 archi- 
tectures with Intel SGX. Chapter 38 covers Intel SGX support for application debug, profiling and performance 
monitoring. 
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Figure 32-1. An Enclave Within the Application’s Virtual Address Space 


32.2 ENCLAVE INTERACTION AND PROTECTION 


Intel SGX allows the protected portion of an application to be distributed in the clear. Before the enclave is built, the 
enclave code and data are free for inspection and analysis. The protected portion is loaded into an enclave where 
its code and data is measured. Once the application’s protected portion of the code and data are loaded into an 
enclave, memory access controls are in place to restrict access by external software. An enclave can prove its iden- 
tity to a remote party and provide the necessary building-blocks for secure provisioning of keys and credentials. 
The application can also request an enclave-specific and platform-specific key that it can use to protect keys and 
data that it wishes to store outside the enclave.! 


1. For additional information, see white papers on Intel SGX at http://software.intel.com/en-us/intel-isa-extensions. 
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Intel SGX introduces two significant capabilities to the Intel Architecture. First is the change in enclave memory 
access semantics. The second is protection of the address mappings of the application. 


32.3 ENCLAVE LIFE CYCLE 


Enclave memory management is divided into two parts: address space allocation and memory commitment. 
Address space allocation is the specification of the range of linear addresses that the enclave may use. This range 
is called the ELRANGE. No actual resources are committed to this region. Memory commitment is the assignment 
of actual memory resources (as pages) within the allocated address space. This two-phase technique allows flexi- 
bility for enclaves to control their memory usage and to adjust dynamically without overusing memory resources 
when enclave needs are low. Commitment adds physical pages to the enclave. An operating system may support 
separate allocate and commit operations. 


During enclave creation, code and data for an enclave are loaded from a clear-text source, i.e. from non-enclave 
memory. 


Untrusted application code starts using an initialized enclave typically by using the EENTER leaf function provided 
by Intel SGX to transfer control to the enclave code residing in the protected Enclave Page Cache (EPC). The 
enclave code returns to the caller via the EEXIT leaf function. Upon enclave entry, control is transferred by hard- 
ware to software inside the enclave. The software inside the enclave switches the stack pointer to one inside the 
enclave. When returning back from the enclave, the software swaps back the stack pointer then executes the 
EEXIT leaf function. 


On processors that support the SGX2 extensions, an enclave writer may add memory to an enclave using the SGX2 
instruction set, after the enclave is built and running. These instructions allow adding additional memory resources 
to the enclave for use in such areas as the heap. In addition, SGX2 instructions allow the enclave to add new 
threads to the enclave. The SGX2 features provide additional capabilities to the software model without changing 
the security properties of the Intel SGX architecture. 


Calling an external procedure from an enclave could be done using the EEXIT leaf function. Software would use 
EEXIT and a software convention between the trusted section and the untrusted section. 


An active enclave consumes resources from the Enclave Page Cache (EPC, see Section 32.5). Intel SGX provides 
the EREMOVE instruction that an EPC manager can use to reclaim EPC pages committed to an enclave. The EPC 
manager uses EREMOVE on every enclave page when the enclave is torn down. After successful execution of 
EREMOVE the EPC page is available for allocation to another enclave. 


32.4 DATASTRUCTURES AND ENCLAVE OPERATION 


There are 2 main data structures associated with operating an enclave, the SGX Enclave Control Structure (SECS, 
see Section 33.7) and the Thread Control Structure (TCS, see Section 33.8). 


There is one SECS for each enclave. The SECS contains meta-data about the enclave which is used by the hardware 
and cannot be directly accessed by software. Included in the SECS is a field that stores the enclave build measure- 
ment value. This field, MRENCLAVE, is initialized by the ECREATE instruction and updated by every EADD and 
EEXTEND. It is locked by EINIT. 


Every enclave contains one or more TCS structures. The TCS contains meta-data used by the hardware to save and 
restore thread specific information when entering/exiting the enclave. There is one field, FLAGS, that may be 
accessed by software. This field can only be accessed by debug enclaves. The flag bit, DBGOPTIN, allows to single 
step into the thread associated with the TCS. (see Section 33.8.1) 


The SECS is created when ECREATE (see Table 32-1) is executed. The TCS can be created using the EADD instruc- 
tion or the SGX2 instructions (see Table 32-2). 


32.5 ENCLAVE PAGE CACHE 


The Enclave Page Cache (EPC) is the secure storage used to store enclave pages when they are a part of an 
executing enclave. For an EPC page, hardware performs additional access control checks to restrict access to the 
page. After the current page access checks and translations are performed, the hardware checks that the EPC page 
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is accessible to the program currently executing. Generally an EPC page is only accessed by the owner of the 
executing enclave or an instruction which is setting up an EPC page 


The EPC is divided into EPC pages. An EPC page is 4KB in size and always aligned on a 4KB boundary. 


Pages in the EPC can either be valid or invalid. Every valid page in the EPC belongs to one enclave instance. Each 
enclave instance has an EPC page that holds its SECS. The security metadata for each EPC page is held in an 
internal micro-architectural structure called Enclave Page Cache Map (EPCM, see Section 32.5.1). 

The EPC is managed by privileged software. Intel SGX provides a set of instructions for adding and removing 
content to and from the EPC. The EPC may be configured by BIOS at boot time. On implementations in which EPC 
memory is part of system DRAM, the contents of the EPC are protected by an encryption engine. 


32.5.1 Enclave Page Cache Map (EPCM) 


The EPC is a secure structure used by the processor to track the contents of the EPC. The EPCM holds one entry 
for each page in the EPC. The format of the EPCM is micro-architectural, and consequently is implementation 
dependent. However, the EPCM contains the following architectural information: 

¢ The status of EPC page with respect to validity and accessibility. 

¢ An SECS identifier (see Section 33.20) of the enclave to which the page belongs. 

° The type of page: regular, SECS, TCS or VA. 

¢ The linear address through which the enclave is allowed to access the page. 

¢ The specified read/write/execute permissions on that page. 

The EPCM structure is used by the CPU in the address-translation flow to enforce access-control on the EPC pages. 


The EPCM structure is described in Table 33-29, and the conceptual access-control flow is described in Section 
33.5. 


The EPCM entries are managed by the processor as part of various instruction flows. 


32.6 ENCLAVE INSTRUCTIONS AND INTEL® SGX 


The enclave instructions available with Intel SGX are organized as leaf functions under three instruction 
mnemonics: ENCLS (ring 0), ENCLU (ring 3), and ENCLV (VT root mode). Each leaf function uses EAX to specify the 
leaf function index, and may require additional implicit input registers as parameters. The use of EAX is implied 
implicitly by the ENCLS, ENCLU, and ENCLV instructions; ModR/M byte encoding is not used with ENCLS, ENCLU, 
and ENCLV. The use of additional registers does not use ModR/M encoding and is implied implicitly by the respective 
leaf function index. 


Each leaf function index is also associated with a unique, leaf-specific mnemonic. A long-form expression of Intel 
SGX instruction takes the form of ENCLx[LEAF_MNEMONIC], where ‘x’ is either ‘S’, ‘U’, or ‘V’. The long-form expres- 
sion provides clear association of the privilege-level requirement of a given “leaf mnemonic”. For simplicity, the 
unique “Leaf_Mnemonic” name is used (omitting the ENCLx for convenience) throughout in this document. 


Details of individual SGX leaf functions are described in Chapter 36. Table 32-1 provides a summary of the instruc- 
tion leaves that are available in the initial implementation of Intel SGX, which is introduced in the 6th generation 
Intel Core processors. Table 32-2 summarizes enhancement of Intel SGX for future Intel processors. 


Table 32-1. Supervisor and User Mode Enclave Instruction Leaf Functions in Long-Form of SGX1 


Supervisor Instruction Description User Instruction Description 

ENCLS[EADD] Add an EPC page to an enclave. ENCLU[EENTER] Enter an enclave. 

ENCLS[EBLOCK] Block an EPC page. ENCLU[EEXIT] Exit an enclave. 

ENCLS[ECREATE] Create an enclave. ENCLU[EGETKEY] Create a cryptographic key. 

ENCLS[EDBGRD] Read data from a debug enclave by debug- | ENCLU[EREPORT] Create a cryptographic report. 
ger. 
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Table 32-1. Supervisor and User Mode Enclave Instruction Leaf Functions in Long-Form of SGX1 


Supervisor Instruction Description User Instruction Description 
ENCLS[EDBGWR] Write data into a debug enclave by debug- | ENCLU[ERESUME] Re-enter an enclave. 
ger. 
ENCLS[EEXTEND] Extend EPC page measurement. 
ENCLS[EINIT] Initialize an enclave. 
ENCLS[ELDB] Load an EPC page in blocked state. 
ENCLS[ELDU] Load an EPC page in unblocked state. 
ENCLS[EPA] Add an EPC page to create a version array. 
ENCLS[EREMOVE] Remove an EPC page from an enclave. 
ENCLS[ETRACK] Activate EBLOCK checks. 
ENCLS[EWB] Write back/invalidate an EPC page. 
Table 32-2. Supervisor and User Mode Enclave Instruction Leaf Functions in Long-Form of SGX2 
Supervisor Instruction Description User Instruction Description 
ENCLS[EAUG] Allocate EPC page to an existing enclave. | ENCLU[EACCEPT] Accept EPC page into the enclave. 
ENCLS[EMODPR] Restrict page permissions. ENCLU[EMODPE] Enhance page permissions. 
ENCLS[EMODT] Modify EPC page type. ENCLU[EACCEPTCOPY] | Copy contents to an augmented EPC 
page and accept the EPC page into 
the enclave. 


Table 32-3. VMX Operation and Supervisor Mode Enclave Instruction Leaf Functions in Long-Form of OVERSUB 


VMX Operation Description Supervisor Instruction Description 
ENCLV[EDECVIRTCHILD] | Decrement the virtual child page count. ENCLS[ERDINFO] Read information about EPC page. 
ENCLV[EINCVIRTCHILD] | Increment the virtual child page count. ENCLS[ETRACKC] Activate EBLOCK checks with conflict 

reporting. 
ENCLV[ESETCONTEXT] | Set virtualization context. ENCLS[ELDBC/UC] Load an EPC page with conflict 
reporting. 
32.7. DISCOVERING SUPPORT FOR INTEL® SGX AND ENABLING ENCLAVE 


INSTRUCTIONS 


Detection of support of Intel SGX and enumeration of available and enabled Intel SGX resources are queried using 

the CPUID instruction. The enumeration interface comprises the following: 

° Processor support of Intel SGX is enumerated by a feature flag in CPUID leaf 07H: CPUID.(EAX=07H, 
ECX=0H):EBX.SGX[bit 2]. If CPUID.(EAX=07H, ECX=0H):EBX.SGX = 1, the processor has support for Intel 
SGX, and requires opt-in enabling by BIOS via IA32_FEATURE_CONTROL MSR. 


If CPUID.(EAX=0 


7H, ECX=0H):EBX.SGX = 1, 


CPUID will 


CPUID.(EAX=12H) on available and/or configured Intel SGX resources. 


report via the available sub-leaves of 


The available and configured Intel SGX resources enumerated by the sub-leaves of CPUID.(EAX=12H) depend 
on the state of BIOS configuration. 
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32.7.1 Intel® SGX Opt-In Configuration 


On processors that support Intel SGX, IA32_FEATURE_CONTROL provides the SGX_ENABLE field (bit 18). Before 
system software can configure and enable Intel SGX resources, BIOS is required to set 
IA32_FEATURE_CONTROL.SGX_ENABLE = 1 to opt-in the use of Intel SGX by system software. 

The semantics of setting SGX_ENABLE follows the rules of IA32_FEATURE_CONTROL.LOCK (bit 0). Software is 
considered to have opted into Intel SGX if and only if IA32_FEATURE_CONTROL.SGX_ENABLE and 
IA32_FEATURE_CONTROL.LOCK are set to 1. The setting of IA32_FEATURE_CONTROL.SGX_ENABLE (bit 18) is not 
reflected by CPUID. 


Table 32-4. Intel® SGX Opt-in and Enabling Behavior 


lee a ae CPUID.(12H) ical ey cae Enclave inciietion 

0 Invalid X X #UD 

1 Valid* X X #UD** 

1 Valid* 0 X #GP 

1 Valid* 1 0 #GP 

1 Valid* 1 1 Available (see Table 32-5 for details 
of SGX1 and SGX2). 

* Leaf 12H enumeration results are dependent on enablement. 

** See list of conditions in the #UD section of the reference pages of ENCLS and ENCLU 


32.7.2 Intel? SGX Resource Enumeration Leaves 


If CPUID.(EAX=07H, ECX=0H):EBX.SGX = 1, the processor also supports querying CPUID with EAX=12H on Intel 
SGX resource capability and configuration. The number of available sub-leaves in leaf 12H depends on the Opt-in 
and system software configuration. Information returned by CPUID.12H is thread specific; software should not 

assume that if Intel SGX instructions are supported on one hardware thread, they are also supported elsewhere. 


A properly configured processor exposes Intel SGX functionality with CPUID.EAX=12H reporting valid information 
(non-zero content) in three or more sub-leaves, see Table 32-5. 


¢ CPUID.(EAX=12H, ECX=0H) enumerates Intel SGX capability, including enclave instruction opcode support. 


¢ CPUID.(EAX=12H, ECX=1H) enumerates Intel SGX capability of processor state configuration and enclave 
configuration in the SECS structure (see Table 33-3). 


¢ CPUID.(EAX=12H, ECX >1) enumerates available EPC resources. 


Table 32-5. CPUID Leaf 12H, Sub-Leaf 0 Enumeration of Intel® SGX Capabilities 


CPUID.(EAX=12H,ECX=0) Description Behavior 
Register Bits 
EAX 0 SGX1: If 1, indicates leaf functions of SGX1 instruction listed in Table 32-1 are supported. 
1 SGX2: If 1, indicates leaf functions of SGX2 instruction listed in Table 32-2 are supported. 
4:2 Reserved (0) 
OVERSUB: If 1, indicates Intel SGX supports instructions: EINCVIRTCHILD, EDECVIRTCHILD, and 
ESETCONTEXT. 
6 OVERSUB: If 1, indicates Intel SGX supports instructions: ETRACKC, ERDINFO, ELDBC, and ELDUC. 
31:7 Reserved (0) 
EBX 31:0 MISCSELECT: Reports the bit vector of supported extended features that can be written to the MISC 
region of the SSA. 
ECX 31:0 Reserved (0). 
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Table 32-5. CPUID Leaf 12H, Sub-Leaf 0 Enumeration of Intel® SGX Capabilities 


CPUID.(EAX=12H,ECX=0) 


Description Behavior 


Register Bits 
7:0 MaxEnclaveSize_Not64: the maximum supported enclave size is 2~(EDX[7:0]) bytes when not in 64-bit 
mode. 
EDX 15:8 MaxEnclaveSize_64: the maximum supported enclave size is 2~(EDX[15:8]) bytes when operating in 64- 
bit mode. 
31:16 Reserved (0). 


Table 32-6. CPUID Leaf 12H, Sub-Leaf 1 Enumeration of Intel® SGX Capabilities 


CPUID.(EAX=12H,ECX=1) 


Description Behavior 


Register Bits 

EAX 31:0 Report the valid bits of SECS.ATTRIBUTES[31:0] that software can set with ECREATE. 
SECS.ATTRIBUTES[n] can be set to 1 using ECREATE only if EAX[n] is 1, where n < 32. 

EBX 31:0 Report the valid bits of SECS.ATTRIBUTES[63:32] that software can set with ECREATE. 
SECS.ATTRIBUTES[n+32] can be set to 1 using ECREATE only if EBX[n] is 1, where n < 32. 

ECX 31:0 Report the valid bits of SECS.ATTRIBUTES[95:64] that software can set with ECREATE. 
SECS.ATTRIBUTES[N+64] can be set to 1 using ECREATE only if ECX[n] is 1, where n < 32. 

EDX 31:0 Report the valid bits of SECS.ATTRIBUTES[1 27:96] that software can set with ECREATE. 


SECS.ATTRIBUTES[n+96] can be set to 1 using ECREATE only if EDX[n] is 1, where n < 32. 


On processors that support Intel SGX1 and SGX2, CPUID leaf 12H sub-leaf 2 report physical memory resources 
available for use with Intel SGX. These physical memory sections are typically allocated by BIOS as Processor 
Reserved Memory, and available to the OS to manage as EPC. 


To enumerate how many EPC sections are available to the EPC manager, software can enumerate CPUID leaf 12H 
with sub-leaf index starting from 2, and decode the sub-leaf-type encoding (returned in EAX[3:0]) until the sub- 
leaf type is invalid. All invalid sub-leaves of CPUID leaf 12H return EAX/EBX/ECX/EDX with 0. 


Table 32-7. CPUID Leaf 12H, Sub-Leaf Index 2 or Higher Enumeration of Intel’ SGX Resources 


CPUID.(EAX=12H,ECX > 1) 


Description Behavior 


Register Bits 
EAX 3:0 OO0O0Ob: This sub-leaf is invalid; EDX:ECX:EBX:EAX return 0. 
0001b: This sub-leaf enumerates an EPC section. EBX:EAX and EDX:ECX provide information on the 
Enclave Page Cache (EPC) section. 
All other encodings are reserved. 
11:4 Reserved (enumerate 0). 
31:12 If EAX[3:0] = 00015, these are bits 31:12 of the physical address of the base of the EPC section. 
Bay 19:0 If EAX[3:0] = 00015, these are bits 51:32 of the physical address of the base of the EPC section. 
31:20 Reserved. 
3:0 If ECX[3:0] = OOOO, then all bits of the EDX:ECX pair are enumerated as O. 
If ECX[3:0] = 00015, then this section has confidentiality and integrity protection. 
If ECX[3:0] = 0010b, then this section has confidentiality protection only. 
ECX All other encodings are reserved. 
11:4 Reserved (enumerate 0). 
31:12 If EAX[3:0] = 00015, these are bits 31:12 of the size of the corresponding EPC section within the 


Processor Reserved Memory. 
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Table 32-7. CPUID Leaf 12H, Sub-Leaf Index 2 or Higher Enumeration of Intel? SGX Resources 


CPUID.(EAX=12H,ECX > 1) Description Behavior 
Register Bits 
EDX 19:0 If EAX[3:0] = 00015, these are bits 51:32 of the size of the corresponding EPC section within the 
Processor Reserved Memory. 
31:20 Reserved. 


32.8 INTEL® SGX INTERACTIONS WITH CONTROL-FLOW ENFORCEMENT 
TECHNOLOGY 


This section discusses extensions to the Intel SGX architecture to support CET. 


32.8.1 CET in Enclaves Model 


Each enclave has its private configuration for CET that is not shared with the CET configurations of the enclosing 
application. On entry into the enclave, the CET state of the enclosing application is saved into scratchpad registers 
inside the processor and the CET state of the enclave is established. On an asynchronous exit, the enclave CET 
state is saved into the enclave state save area frame. On exit from the enclave, the CET state of the enclosing 
application is re-established from the scratchpad registers. 


A new page type, PT_SS_FIRST, is used to denote pages in an enclave that can be used as a first page of a shadow 
stack. 


A new page type, PT_SS_REST, is used to denote pages in an enclave that can be used as a non-first page of a 
shadow stack. 


A page denoted as PT_SS_FIRST and PT_SS_REST will be a legal target for shadow_stack_load, 
shadow_stack_store, and regular load operations. Regular stores will be disallowed to such pages. A 
PT_SS_FIRST/PT_SS_REST page must be writeable in the IA page tables and in EPT. 


When in enclave mode, shadow_stack_load and shadow_stack_store operations must be to addresses in the 
enclave ELRANGE. 


The EAUG instruction is extended to allocate pages of type PT_SS_FIRST/PT_SS_REST; this page type requires 
specifying a SECINFO structure with page parameters. Shadow page permission must be R/W. Regular R/W pages 
may continue to be allocated by providing a SECINFO pointer value of 0. Regular R/W pages may also be allocated 
by providing a SECINFO structure that specifies the page parameters. The EAUG instruction creates a shadow- 
stack-restore token at offset OxFF8 on a PT_SS_FIRST page. This allows a dynamically created shadow stack to be 
restored using the RSTORSSP instruction. The EADD and EAUG instructions disallow creation of a PT_SS_FIRST or 
PT_SS_REST page as the first or last page in ELRANGE. 


The EADD instruction requires that the PT_SS_REST page be all zeroes. The EADD instruction requires that a 
PT_SS_FIRST page be all zeroes except the 8 bytes at offset OxFF8 on that page that must have a shadow-stack- 
restore token. This shadow-stack-restore token must have a linear address which is the linear address of the 
PT_SS_FIRST page + 4096. As an enclave could be loaded at varying linear addresses, the enclave builder should 
not extend the measurement of the PT_SS_FIRST pages into the measurement registers. On first entry on to the 
enclave using a TCS, the enclave software can use the RSTORSSP instruction to restore its SSP. Subsequent to 
performing a RSTORSSP, the enclave software can use the INCSSP instruction to pop the previous-ssp token that 
is created by the RSTORSSP instruction at the top of the restored shadow stack. 


On an enclave entry, the SSP will be initialized to the value in a new TCS field called PREVSSP. The PREVSSP field is 
written with the value of SSP on enclave exit and is loaded into SSP at enclave entry. When a TCS page is added 
using EADD or accepted using EACCEPT, the processor requires the PREVSSP field to be initialized to 0. 
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32.8.2 Operations Not Supported on Shadow Stack Pages 

The following operations are not allowed on pages of type PT_SS_FIRST and PT_SS_REST: 
¢ EACCEPTCOPY 
¢ EMODPR 
¢ EMODPE 


32.8.3 Indirect Branch Tracking - Legacy Compatibility Treatment 


The legacy code page bitmap is tested using the page offset within the ELRANGE instead of the absolute linear 
address of the address where ENDBRANCH was missed; see the detailed algorithm in Section 18.3.6, “Legacy 
Compatibility Treatment” in the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1 for 
additional details. 
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CHAPTER 33 
ENCLAVE ACCESS CONTROL AND DATA STRUCTURES 


33.1 OVERVIEW OF ENCLAVE EXECUTION ENVIRONMENT 


When an enclave is created, it has a range of linear addresses to which the processor applies enhanced access 
control. This range is called the ELRANGE (see Section 32.3). When an enclave generates a memory access, the 
existing IA32 segmentation and paging architecture are applied. Additionally, linear addresses inside the ELRANGE 
must map to an EPC page otherwise when an enclave attempts to access that linear address a fault is generated. 


The EPC pages need not be physically contiguous. System software allocates EPC pages to various enclaves. 
Enclaves must abide by OS/VMM imposed segmentation and paging policies. OS/VMM-managed page tables and 
extended page tables provide address translation for the enclave pages. Hardware requires that these pages are 
properly mapped to EPC (any failure generates an exception). 


Enclave entry must happen through specific enclave instructions: 

¢ ENCLU[EENTER], ENCLU[ERESUME]. 

Enclave exit must happen through specific enclave instructions or events: 
¢ ENCLU[EEXIT], Asynchronous Enclave Exit (AEX). 


Attempts to execute, read, or write to linear addresses mapped to EPC pages when not inside an enclave will result 
in the processor altering the access to preserve the confidentiality and integrity of the enclave. The exact behavior 
may be different between implementations. As an example a read of an enclave page may result in the return of all 
one's or return of cyphertext of the cache line. Writing to an enclave page may result in a dropped write ora 
machine check at a later time. The processor will provide the protections as described in Section 33.4 and Section 
33.5 on such accesses. 


33.2 TERMINOLOGY 
A memory access to the ELRANGE and initiated by an instruction executed by an enclave is called a Direct Enclave 
Access (Direct EA). 


Memory accesses initiated by certain Intel® SGX instruction leaf functions such as ECREATE, EADD, EDBGRD, 
EDBGWR, ELDU/ELDB, EWB, EREMOVE, EENTER, and ERESUME to EPC pages are called Indirect Enclave Accesses 
(Indirect EA). Table 33-1 lists additional details of the indirect EA of SGX1 and SGX2 extensions. 


Direct EAs and Indirect EAs together are called Enclave Accesses (EAs). 
Any memory access that is not an Enclave Access is called a non-enclave access. 


33.3. ACCESS-CONTROL REQUIREMENTS 


Enclave accesses have the following access-control attributes: 
¢ All memory accesses must conform to segmentation and paging protection mechanisms. 
* Code fetches from inside an enclave to a linear address outside that enclave result in a #GP(0) exception. 


¢ Shadow-stack-load or shadow-stack-store from inside an enclave to a linear address outside that enclave 
results in a #GP(0) exception. 


° Non-enclave accesses to EPC memory result in undefined behavior. EPC memory is protected as described in 
Section 33.4 and Section 33.5 on such accesses. 


° EPC pages of page types PT_REG, PT_TCS and PT_TRIM must be mapped to ELRANGE at the linear address 
specified when the EPC page was allocated to the enclave using ENCLS[EADD] or ENCLS[EAUG] leaf functions. 
Enclave accesses through other linear address result in a #PF with the PFEC.SGX bit set. 
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* Direct EAs to any EPC pages must conform to the currently defined security attributes for that EPC page in the 
EPCM. These attributes may be defined at enclave creation time (EADD) or when the enclave sets them using 
SGX2 instructions. The failure of these checks results in a #PF with the PFEC.SGX bit set. 


— Target page must belong to the currently executing enclave. 

— Data may be written to an EPC page if the EPCM allow write access. 

— Data may be read from an EPC page if the EPCM allow read access. 

— Instruction fetches from an EPC page are allowed if the EPCM allows execute access. 


— Shadow-stack-load from an EPC page and shadow-stack-store to an EPC page are allowed only if the page 
type is PT_SS_FIRST or PT_SS_REST. 


— Data writes that are not shadow-stack-store are not allowed if the EPCM page type is PT_SS_FIRST or 
PT_SS_REST. 


— Target page must not have a restricted page type! (PT_SECS, PT_TCS, PT_VA, or PT_TRIM). 
— The EPC page must not be BLOCKED. 

— The EPC page must not be PENDING. 

— The EPC page must not be MODIFIED. 


33.4 SEGMENT-BASED ACCESS CONTROL 


Intel SGX architecture does not modify the segment checks performed by a logical processor. All memory accesses 
arising from a logical processor in protected mode (including enclave access) are subject to segmentation checks 
with the applicable segment register. 


To ensure that outside entities do not modify the enclave's logical-to-linear address translation in an unexpected 
fashion, ENCLU[EENTER] and ENCLU[ERESUME] check that CS, DS, ES, and SS, if usable (i.e., not null), have 
segment base value of zero. A non-zero segment base value for these registers results in a #GP(0). 


On enclave entry either via EENTER or ERESUME, the processor saves the contents of the external FS and GS regis- 
ters, and loads these registers with values stored in the TCS at build time to enable the enclave’s use of these regis- 
ters for accessing the thread-local storage inside the enclave. On EEXIT and AEX, the contents at time of entry are 
restored. On AEX, the values of FS and GS are saved in the SSA frame. On ERESUME, FS and GS are restored from 
the SSA frame. The details of these operations can be found in the descriptions of EENTER, ERESUME, EEXIT, and 
AEX flows. 


33.5 PAGE-BASED ACCESS CONTROL 


33.5.1. Access-control for Accesses that Originate from non-SGX Instructions 


Intel SGX builds on the processor's paging mechanism to provide page-granular access-control for enclave pages. 
Enclave pages are designed to be accessible only from inside the currently executing enclave if they belong to that 
enclave. In addition, enclave accesses must conform to the access control requirements described in Section 33.3. 
or through certain Intel SGX instructions. Attempts to execute, read, or write to linear addresses mapped to EPC 

pages when not inside an enclave will result in the processor altering the access to preserve the confidentiality and 
integrity of the enclave. The exact behavior may be different between implementations. 


33.5.2 Memory Accesses that Split across ELRANGE 


Memory data accesses are allowed to split across ELRANGE (i.e., a part of the access is inside ELRANGE and a part 
of the access is outside ELRANGE) while the processor is inside an enclave. If an access splits across ELRANGE, the 


1. EPCM may allow write, read or execute access only for pages with page type PT_REG. 


33-2 Vol. 3D 


ENCLAVE ACCESS CONTROL AND DATA STRUCTURES 


processor splits the access into two sub-accesses (one inside ELRANGE and the other outside ELRANGE), and each 
access is evaluated. A code-fetch access that splits across ELRANGE results in a #GP due to the portion that lies 
outside of the ELRANGE. 


33.5.3. Implicit vs. Explicit Accesses 


Memory accesses originating from Intel SGX instruction leaf functions are categorized as either explicit accesses or 
implicit accesses. Table 33-1 lists the implicit and explicit memory accesses made by Intel SGX leaf functions. 


33.5.3.1 Explicit Accesses 


Accesses to memory locations provided as explicit operands to Intel SGX instruction leaf functions, or their linked 
data structures are called explicit accesses. 


Explicit accesses are always made using logical addresses. These accesses are subject to segmentation, paging, 
extended paging, and APIC-virtualization checks, and trigger any faults/exit associated with these checks when the 
access is made. 


The interaction of explicit memory accesses with data breakpoints is leaf-function-specific, and is documented in 
Section 38.3.4. 


33.5.3.2 Implicit Accesses 


Accesses to data structures whose physical addresses are cached by the processor are called implicit accesses. 
These addresses are not passed as operands of the instruction but are implied by use of the instruction. 


These accesses do not trigger any access-control faults/exits or data breakpoints. Table 33-1 lists memory objects 
that Intel SGX instruction leaf functions access either by explicit access or implicit access. The addresses of explicit 
access objects are passed via register operands with the second through fourth column of Table 33-1 matching 
implicitly encoded registers RBX, RCX, RDX. 


Physical addresses used in different implicit accesses are cached via different instructions and for different dura- 
tions. The physical address of SECS associated with each EPC page is cached at the time the page is added to the 
enclave via ENCLS[EADD] or ENCLS[EAUG], or when the page is loaded to EPC via ENCLS[ELDB] or ENCLS[ELDU]. 
This binding is severed when the corresponding page is removed from the EPC via ENCLS[EREMOVE] or 
ENCLS[EWB]. Physical addresses of TCS and SSA pages are cached at the time of most-recent enclave entry. Exit 
from an enclave (ENCLU[EEXIT] or AEX) flushes this caching. Details of Asynchronous Enclave Exit is described in 
Chapter 35. 


The physical addresses that are cached for use by implicit accesses are derived from logical (or linear) addresses 
after checks such as segmentation, paging, EPT, and APIC virtualization checks. These checks may trigger excep- 
tions or VM exits. Note, however, that such exception or VM exits may not occur after a physical address is cached 
and used for an implicit access. 


Table 33-1. List of Implicit and Explicit Memory Access by Intel® SGX Enclave Instructions 


Instr. Leaf Enum. Explicit 1 Explicit 2 Explicit 3 Implicit 
EACCEPT SGX2 SECINFO EPCPAGE SECS 
EACCEPTCOPY SGX2 SECINFO EPCPAGE (Src) EPCPAGE (Dst) 

EADD SGX1 PAGEINFO and linked structures EPCPAGE 

EAUG SGX2 PAGEINFO and linked structures EPCPAGE SECS 
EBLOCK SGX1 EPCPAGE SECS 
ECREATE SGX1 PAGEINFO and linked structures EPCPAGE 

EDBGRD SGX1 EPCADDR Destination SECS 
EDBGWR SGX1 EPCADDR Source SECS 
EDECVIRTCHILD OVERSUB EPCPAGE SECS 

EENTER SGX1 TCS and linked SSA SECS 
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Table 33-1. List of Implicit and Explicit Memory Access by Intel® SGX Enclave Instructions (Contd.) 


Instr. Leaf Enum. Explicit 1 Explicit 2 Explicit 3 Implicit 
EEXIT SGX1 SECS, TCS 
EEXTEND SGX1 SECS EPCPAGE 
EGETKEY SGX1 KEYREQUEST KEY SECS 
EINCVIRTCHILD OVERSUB EPCPAGE SECS 
EINIT SGX1 SIGSTRUCT SECS EINITTOKEN 
ELDB/ELDU SGX1 PAGEINFO and linked structures, PCMD EPCPAGE VAPAGE 
ELDBC/ELDUC OVERSUB PAGEINFO and linked structures EPCPAGE VAPAGE 
EMODPE SGX2 SECINFO EPCPAGE 
EMODPR SGX2 SECINFO EPCPAGE SECS 
EMODT SGX2 SECINFO EPCPAGE SECS 
EPA SGX1 EPCADDR 
ERDINFO OVERSUB RDINFO EPCPAGE 
EREMOVE SGX1 EPCPAGE SECS 
EREPORT SGX1 TARGETINFO REPORTDATA OUTPUTDATA SECS 
ERESUME SGX1 TCS and linked SSA SECS 
ESETCONTEXT OVERSUB SECS ContextValue 
ETRACK SGX1 EPCPAGE 
ETRACKC OVERSUB EPCPAGE 
EWB SGX1 PAGEINFO and linked structures, PCMD EPCPAGE VAPAGE SECS 
Asynchronous Enclave Exit* SECS, TCS, 

SSA 


*Details of Asynchronous Enclave Exit (AEX) is described in Section 35.4 


33.6 INTEL® SGX DATA STRUCTURES OVERVIEW 


Enclave operation is managed via a collection of data structures. Many of the top-level data structures contain sub- 
structures. The top-level data structures relate to parameters that may be used in enclave setup/maintenance, by 
Intel SGX instructions, or AEX event. The top-level data structures are: 


® SGX Enclave Control Structure (SECS) 
¢ Thread Control Structure (TCS) 

¢ State Save Area (SSA) 

¢ Page Information (PAGEINFO) 

* Security Information (SECINFO) 

¢ Paging Crypto MetaData (PCMD) 

° Enclave Signature Structure (SIGSTRUCT) 
¢  EINIT Token Structure (EINITTOKEN) 
® Report Structure (REPORT) 

¢ Report Target Info (TARGETINFO) 

° Key Request (KEYREQUEST) 

°® Version Array (VA) 

°® Enclave Page Cache Map (EPCM) 

* Read Info (RDINFO) 
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Details of the top-level data structures and associated sub-structures are listed in Section 33.7 through Section 


33.20. 


33.7 


SGX ENCLAVE CONTROL STRUCTURE (SECS) 


The SECS data structure requires 4K-Bytes alignment. 


Table 33-2. Layout of SGX Enclave Control Structure (SECS) 


Field OFFSET (Bytes) Size (Bytes) Description 

SIZE 0 8 Size of enclave in bytes; must be power of 2. 

BASEADDR 8 8 Enclave Base Linear Address must be naturally aligned to size. 

SSAFRAMESIZE 16 4 Size of one SSA frame in pages, including XSAVE, pad, GPR, and MISC (if 
CPUID.(EAX=12H, ECX=0):.EBX != 0). 

MISCSELECT 20 4 Bit vector specifying which extended features are saved to the MISC region 
(see Section 33.7.2) of the SSA frame when an AEX occurs. 

CET_LEG_BITMAP | 24 8 Page aligned offset of legacy code page bitmap from enclave base. Soft- 

_OFFSET ware is expected to program this offset such that the entire bitmap re- 
sides in the ELRANGE when legacy compatibility mode for indirect branch 
tracking is enabled. However this is not enforced by the hardware. 

This field exists when CPUID.(EAX=7, ECX=0):EDX.CET_IBT[bit 20] is enu- 
merated as 1, else it is reserved. 

CET_ATTRIBUTES | 32 1 CET feature attributes of the enclave; see Table 33-5. This field exists 
when CPUID.(EAX=1 2,ECX=1):EAX[6] is enumerated as 1, else it is 
reserved. 

RESERVED 24 24 

ATTRIBUTES 48 16 Attributes of the Enclave, see Table 33-3. 

MRENCLAVE 64 32 Measurement Register of enclave build process. See SIGSTRUCT for format. 

RESERVED 96 32 

MRSIGNER 128 32 Measurement Register extended with the public key that verified the 
enclave. See SIGSTRUCT for format. 

RESERVED 160 32 

CONFIGID 192 64 Post EINIT configuration identity. 

ISVPRODID 256 2 Product ID of enclave. 

ISVSVN 258 2 Security version number (SVN) of the enclave. 

CONFIGSVN 260 2 Post EINIT configuration security version number (SVN). 

RESERVED 262 3834 The RESERVED field consists of the following: 


« EID: An 8 byte Enclave Identifier. Its location is implementation specific. 

* PAD: A 352 bytes padding pattern from the Signature (used for key 
derivation strings). It's location is implementation specific. 

* VIRTCHILDCNT: An 8 byte Count of virtual children that have been paged 
out by a VMM. Its location is implementation specific. 

= ENCLAVECONTEXT: An 8 byte Enclave context pointer. Its location is 
implementation specific. 

* ISVFAMILYID: A 16 byte value assigned to identify the family of products 
the enclave belongs to. 

= ISVEXTPRODID: A 16 byte value assigned to identify the product 
identity of the enclave. 

« The remaining 3226 bytes are reserved area. 

The entire 3834 byte field must be cleared prior to executing ECREATE. 
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33.7.1 ATTRIBUTES 


The ATTRIBUTES data structure is comprised of bit-granular fields that are used in the SECS, the REPORT and the 
KEYREQUEST structures. CPUID.(EAX=12H, ECX=1) enumerates a bitmap of permitted 1-setting of bits in ATTRI- 
BUTES. 


Table 33-3. Layout of ATTRIBUTES Structure 


Field Bit Position Description 

INIT 0 This bit indicates if the enclave has been initialized by EINIT. It must be cleared when loaded as 
part of ECREATE. For EREPORT instruction, TARGET_INFO.ATTRIBUTES[ENIT] must always be 1 to 
match the state after EINIT has initialized the enclave. 

DEBUG 1 If 1, the enclave permit debugger to read and write enclave data using EDBGRD and EDBGWR. 

MODE64BIT 2 Enclave runs in 64-bit mode. 

RESERVED 3 Must be Zero. 

PROVISIONKEY 4 Provisioning Key is available from EGETKEY. 

EINITTOKEN_KEY |5 EINIT token key is available from EGETKEY. 

CET 6 Enable CET attributes. When CPUID.(EAX=1 2H, ECX=1):EAX[6] is O this bit is reserved and must be 
0. 

KSS 7 Key Separation and Sharing Enabled. 

RESERVED 63:8 Must be zero. 

XFRM 127:64 XSAVE Feature Request Mask. See Section 37.7. 


33.7.2 SECS.MISCSELECT Field 


CPUID.(EAX=12H, ECX=0):EBX[31:0] enumerates which extended information that the processor can save into 
the MISC region of SSA when an AEX occurs. An enclave writer can specify via SIGSTRUCT how to set the 
SECS.MISCSELECT field. The bit vector of MISCSELECT selects which extended information is to be saved in the 
MISC region of the SSA frame when an AEX is generated. The bit vector definition of extended information is listed 


in Table 33-4. 
If CPUID.(EAX=12H, ECX=0):EBX[31:0] = 0, SECS.MISCSELECT field must be all zeros. 
The SECS.MISCSELECT field determines the size of MISC region of the SSA frame, see Section 33.9.2. 


Table 33-4. Bit Vector Layout of MISCSELECT Field of Extended Information 


Field Bit Position Description 
EXINFO 0 Report information about page fault and general protection exception that occurred inside an 
enclave. 
CPINFO 1 Report information about control protection exception that occurred inside an enclave. When 
CPUID.(EAX=1 2H, ECX=0):EBX[1] is 0, this bit is reserved. 
Reserved 31:2 Reserved (0). 


33.7.3 SECS.CET_ATTRIBUTES Field 


The SECS.CET_ATTRIBUTES field can be used by the enclave writer to enable various CET attributes in an enclave. 
This field exists when CPUID.(EAX=12,ECX=1):EAX[6] is enumerated as 1. Bits 1:0 are defined when 
CPUID.(EAX=7, ECX=0):ECX.CET_SS is 1, and bits 5:2 are defined when CPUID.(EAX=7, ECX=0):EDX.CET_IBT is 
1, 
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Table 33-5. Bit Vector Layout of CET_ATTRIBUTES Field of Extended Information 


Field Bit Position Description 
SH_STK_EN 0 When set to 1, enable shadow stacks. 
WR_SHSTK_EN 1 When set to 1, enables the WRSS{D,Q}W instructions. 
ENDBR_EN 2 When set to 1, enables indirect branch tracking. 
LEG_IW_EN 3 Enable legacy compatibility treatment for indirect branch tracking. 
NO_TRACK_EN 4 When set to 1, enables use of no-track prefix for indirect branch tracking. 
SUPPRESS_DIS 5 When set to 1, disables suppression of CET indirect branch tracking on legacy compatibility. 
Reserved 7:6 Reserved (0). 


33.8 


THREAD CONTROL STRUCTURE (TCS) 


Each executing thread in the enclave is associated with a Thread Control Structure. It requires 4K-Bytes alignment. 


Table 33-6. Layout of Thread Control Structure (TCS) 


Field OFFSET (Bytes) | Size (Bytes) Description 

STAGE 0 8 Enclave execution state of the thread controlled by this TCS. A value of 0 indi- 
cates that this TCS is available for enclave entry. A value of 1 indicates that a 
processer is currently executing an enclave in the context of this TCS. 

FLAGS 8 8 The thread's execution flags (see Section 33.8.1). 

OSSA 16 8 Offset of the base of the State Save Area stack, relative to the enclave base. 
Must be page aligned. 

CSSA 24 4 Current slot index of an SSA frame, cleared by EADD and EACCEPT. 

NSSA 28 4 Number of available slots for SSA frames. 

OENTRY 32 8 Offset in enclave to which control is transferred on EENTER relative to the 
base of the enclave. 

AEP 40 8 The value of the Asynchronous Exit Pointer that was saved at EENTER time. 

OFSBASE 48 8 Offset to add to the base address of the enclave for producing the base 
address of FS segment inside the enclave. Must be page aligned. 

OGSBASE 56 8 Offset to add to the base address of the enclave for producing the base 
address of GS segment inside the enclave. Must be page aligned. 

FSUMIT 64 4 Size to become the new FS limit in 32-bit mode. 

GSLIMIT 68 4 Size to become the new GS limit in 32-bit mode. 

OCETSSA 72 8 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, this field provides the offset of 
the CET state save area from enclave base. When CPUID.(EAX=1 2H, 
ECX=1):EAX[6] is 0, this field is reserved and must be 0. 

PREVSSP 80 8 When CPUID.(EAX=07H, ECX=00h):ECX[CET_SS] is 1, this field records the SSP 
at the time of AEX or EEXIT; used to setup SSP on entry. When 
CPUID.(EAX=07H, ECX=00h):ECX[CET_SS] is 0, this field is reserved and must 
be 0. 

RESERVED 72 4024 Must be zero. 
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33.8.1 TCS.FLAGS 


Table 33-7. Layout of TCS.FLAGS Field 


Field Bit Position Description 


DBGOPTIN 0 If set, allows debugging features (single-stepping, breakpoints, etc.) to be enabled and active while 
executing in the enclave on this TCS. Hardware clears this bit on EADD. A debugger may later mod- 
ify it if the enclave’s ATTRIBUTES.DEBUG is set. 


RESERVED 63:1 


33.8.2 State Save Area Offset (OSSA) 


The OSSA points to a stack of State Save Area (SSA) frames (see Section 33.9) used to save the processor state 
when an interrupt or exception occurs while executing in the enclave. 


33.8.3. Current State Save Area Frame (CSSA) 


CSSA is the index of the current SSA frame that will be used by the processor to determine where to save the 
processor state on an interrupt or exception that occurs while executing in the enclave. It is an index into the array 
of frames addressed by OSSA. CSSA is incremented on an AEX and decremented on an ERESUME. 


33.8.4 Number of State Save Area Frames (NSSA) 


NSSA specifies the number of SSA frames available for this TCS. There must be at least one available SSA frame 
when EENTER-ing the enclave or the EENTER will fail. 


33.9 STATE SAVE AREA (SSA) FRAME 


When an AEX occurs while running in an enclave, the architectural state is saved in the thread’s current SSA frame, 
which is pointed to by TCS.CSSA. An SSA frame must be page aligned, and contains the following regions: 


¢ The XSAVE region starts at the base of the SSA frame, this region contains extended feature register state in 
an XSAVE/FXSAVE-compatible non-compacted format. 


¢ A Pad region: software may choose to maintain a pad region separating the XSAVE region and the MISC region. 
Software choose the size of the pad region according to the sizes of the MISC and GPRSGxX regions. 


° The GPRSGX region. The GPRSGX region is the last region of an SSA frame (see Table 33-8). This is used to hold 
the processor general purpose registers (RAX ... R15), the RIP, the outside RSP and RBP, RFLAGS and the AEX 
information. 


¢ The MISC region (If CPUIDEAX=12H, ECX=0):EBX[31:0] != 0). The MISC region is adjacent to the GRPSGX 
region, and may contain zero or more components of extended information that would be saved when an AEX 
occurs. If the MISC region is absent, the region between the GPRSGX and XSAVE regions is the pad region that 
software can use. If the MISC region is present, the region between the MISC and XSAVE regions is the pad 
region that software can use. See additional details in Section 33.9.2. 


Table 33-8. Top-to-Bottom Layout of an SSA Frame 
Region | Offset (Byte) Size (Bytes) Description 
XSAVE 0 Calculate using CPUID | The size of XSAVE region in SSA is derived from the enclave's support of the col- 
leaf ODH information | lection of processor extended states that would be managed by XSAVE. The 
enablement of those processor extended state components in conjunction with 
CPUID leaf ODH information determines the XSAVE region size in SSA. 


Pad End of XSAVE ‘| Chosen by enclave Ensure the end of GPRSGX region is aligned to the end of a 4KB page. 
region writer 
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Region | Offset (Byte) Size (Bytes) Description 
MISC base of GPRSGX | Calculate from high- | See Section 33.9.2. 
- sizeof(MISC) _| est set bit of 
SECS.MISCSELECT 
GPRSGX |SSAFRAMESIZE | 176 See Table 33-9 for layout of the GPRSGX region. 
-176 
33.9.1 GPRSGX Region 


The layout of the GPRSGX region is shown in Table 33-9. 


Table 33-9. Layout of GPRSGX Portion of the State Save Area 


Field OFFSET (Bytes) | Size (Bytes) Description 
RAX 0 8 
RCX 8 8 
RDX 16 8 
RBX 24 8 
RSP 32 8 
RBP 40 8 
RSI 48 8 
RDI 56 8 
R8 64 8 
R9 72 8 
R10 80 8 
R11 88 8 
R12 96 8 
R13 104 8 
R14 112 8 
R15 120 8 
RFLAGS 128 8 Flag register. 
RIP 136 8 Instruction pointer. 
URSP 144 8 Non-Enclave (outside) stack pointer. Saved by EENTER, restored on AEX. 
URBP 152 8 Non-Enclave (outside) RBP pointer. Saved by EENTER, restored on AEX. 
EXITINFO 160 4 Contains information about exceptions that cause AEXs, which might be 
needed by enclave software (see Section 33.9.1.1). 
RESERVED 164 4 
FSBASE 168 8 FS BASE. 
GSBASE 176 8 GS BASE. 
33.9.1.1 EXITINFO 


EXITINFO contains the information used to report exit reasons to software inside the enclave. It is a 4 byte field laid 
out as in Table 33-10. The VALID bit is set only for the exceptions conditions which are reported inside an enclave. 
See Table 33-11 for which exceptions are reported inside the enclave. If the exception condition is not one reported 
inside the enclave then VECTOR and EXIT_TYPE are cleared. 
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When a higher priority event, such as SMI, and a pending debug exception occur at the same time when executing 
inside an enclave, the higher priority event has precedence. As an example for an SMI, the SSA exit info is zero. 
The debug exception will be delivered upon return from the SMI. In such cases, the EXITINFO field will not contain 
the information of a debug exception. 


Table 33-10. Layout of EXITINFO Field 


Field Bit Position Description 
VECTOR 7:0 Exception number of exceptions reported inside enclave. 
EXIT_TYPE 10:8 011b: Hardware exceptions. 


110b: Software exceptions. 
Other values: Reserved. 


RESERVED 30:11 Reserved as zero. 


VALID 31 O: unsupported exceptions. 
1: Supported exceptions. Includes two categories: 


¢ —_Unconditionally supported exceptions: #DE, #DB, #BP, #BR, #UD, #MF, #AC, #XM. 
* ~~ Conditionally supported exception: 

—  #PF, #GP if SECS.MISCSELECT.EXINFO = 1. 

— #CP if SECS.MISCSELECT.CPINFO=1. 


33.9.1.2 VECTOR Field Definition 


Table 33-11 contains the VECTOR field. This field contains information about some exceptions which occur inside 
the enclave. These vector values are the same as the values that would be used when vectoring into regular excep- 
tion handlers. All values not shown are not reported inside an enclave. 


Table 33-11. Exception Vectors 


Name Vector # Description 
#DE 0 Divider exception. 
#DB 1 Debug exception. 
#BP 3 Breakpoint exception. 
#BR 5 Bound range exceeded exception. 
#UD 6 Invalid opcode exception. 
#GP 13 General protection exception. Only reported if SECS.MISCSELECT.EXINFO = 1. 
#PF 14 Page fault exception. Only reported if SECS.MISCSELECT.EXINFO = 1. 
#MF 16 x87 FPU floating-point error. 
#AC 17 Alignment check exceptions. 
#XM 19 SIMD floating-point exceptions. 
#CP 21 Control protection exception. Only reported if SECS.MISCSELECT.CPINFO=1. 


33.9.2 MISC Region 


The layout of the MISC region is shown in Table 33-12. The number of components that the processor supports in 
the MISC region corresponds to the bits of CPUID.(EAX=12H, ECX=0):EBX[31:0] set to 1. Each set bit in 
CPUID.(EAX=12H, ECX=0):EBX[31:0] has a defined size for the corresponding component, as shown in Table 
33-12. Enclave writers needs to do the following: 


* Decide which MISC region components will be supported for the enclave. 
e Allocate an SSA frame large enough to hold the components chosen above. 
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¢ Instruct each enclave builder software to set the appropriate bits in SECS.MISCSELECT. 


The first component, EXINFO, starts next to the GPRSGX region. Additional components in the MISC region grow 
in ascending order within the MISC region towards the XSAVE region. 


The size of the MISC region is calculated as follows: 
° = If CPUID.(EAX=12H, ECX=0):EBX[31:0] = 0, MISC region is not supported. 


e = If CPUID.(EAX=12H, ECX=0):EBX[31:0] != 0, the size of MISC region is derived from sum of the highest bit set 
in SECS.MISCSELECT and the size of the MISC component corresponding to that bit. Offset and size 
information of currently defined MISC components are listed in Table 33-12. For example, if the highest bit set 
in SECS.MISCSELECT is bit 0, the MISC region offset is OFFSET(GPRSGX)-16 and size is 16 bytes. 


¢ The processor saves a MISC component i in the MISC region if and only if SECS.MISCSELECT[i] is 1. 


Table 33-12. Layout of MISC region of the State Save Area 


MISC Components | OFFSET (Bytes) | Size (Bytes) Description 


EXINFO Offset(GPRSGX) -16 | 16 If CPUID.(EAX=1 2H, ECX=0):EBX[0] = 1, exception information on #GP or 
#PF that occurred inside an enclave can be written to the EXINFO structure 
if specified by SECS.MISCSELECT[O] = 1. 

If CPUID.(EAX=1 2H, ECX=0):EBX[1] = 1, exception information on #CP that 
occurred inside an enclave can be written to the EXINFO structure if speci- 
fied by SECS.MISCSELECT[1] = 1. 


Future Extension | Below EXINFO TBD Reserved. (Zero size if CPUID.(EAX=12H, ECX=0):€BX[31:1] =0). 


33.9.2.1  EXINFO Structure 
Table 33-13 contains the layout of the EXINFO structure that provides additional information. 


Table 33-13. Layout of EXINFO Structure 
Field OFFSET (Bytes) | Size (Bytes) Description 


MADDR 0 8 If #PF: contains the page fault linear address that caused a page fault. 
If #GP: the field is cleared. 
If #CP: the field is cleared. 


ERRCD 8 4 Exception error code for either #GP or #PF. 
RESERVED 12 


33.9.2.2 Page Fault Error Code 
Table 33-14 contains page fault error code that may be reported in EXINFO.ERRCD. 


Table 33-14. Page Fault Error Code 


Name Bit Position Description 
P 0 Same as non-SGX page fault exception P flag. 
W/R 1 Same as non-SGX page fault exception W/R flag. 
u/s! 2 Always set to 1 (user mode reference). 
RSVD 3 Same as non-SGX page fault exception RSVD flag. 
\/D 4 Same as non-SGX page fault exception I/D flag. 
PK 5 Protection Key induced fault. 
RSVD 14:6 Reserved. 
SGX 15 EPCM induced fault. 
RSVD 31:5 Reserved. 
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NOTES: 
1. Page faults incident to enclave mode that report U/S=0 are not reported in EXINFO. 


33.10 CET STATE SAVE AREA FRAME 


The CET state save area consists of an array of CET state save frames. The number of CET state save frames is 
equal to the TCS.NSSA. The current CET SSA frame is indicated by TCS.CSSA. The offset of the CET state save area 
is specified by TCS.OCETSSA. 


Table 33-15. Layout of CET State Save Area Frame 


Offset 


Field (Bytes) Size (Bytes) Description 
SSP 0 8 Shadow Stack Pointer. 
This field is reserved when CPUID.(EAX=7, ECX=0):ECX[CET_SS] is 0. 
IB_TRACK_STATE |8 8 Indirect branch tracker state: 


Bit O: SUPPRESS - suppressed(1), tracking(0) 

Bit 1: TRACKER - IDLE (0), WAIT_FOR_ENDBRANCH (1) 

Bits 63:2 - Reserved 

This field is reserved when CPUID.(EAX=7, ECX=0):EDX[CET_IBT] is 0. 


33.11 PAGE INFORMATION (PAGEINFO) 


PAGEINFO is an architectural data structure that is used as a parameter to the EPC-management instructions. It 
requires 32-Byte alignment. 


Table 33-16. Layout of PAGEINFO Data Structure 


Field OFFSET (Bytes) | Size (Bytes) Description 
LINADDR 0 8 Enclave linear address. 
SRCPGE 8 8 Effective address of the page where contents are located. 
SECINFO/PCMD | 16 8 Effective address of the SECINFO or PCMD (for ELDU, ELDB, EWB) structure for 
the page. 
SECS 24 8 Effective address of EPC slot that currently contains the SECS. 


33.12 SECURITY INFORMATION (SECINFO) 


The SECINFO data structure holds meta-data about an enclave page. 


Table 33-17. Layout of SECINFO Data Structure 


Field OFFSET (Bytes) | Size (Bytes) Description 
FLAGS 0 8 Flags describing the state of the enclave page. 
RESERVED 8 56 Must be zero. 


33.12.1 SECINFO.FLAGS 
The SECINFO.FLAGS are a set of fields describing the properties of an enclave page. 


33-12 Vol. 3D 


ENCLAVE ACCESS CONTROL AND DATA STRUCTURES 


Table 33-18. Layout of SECINFO.FLAGS Field 


Field Bit Position Description 

R 0 If 1 indicates that the page can be read from inside the enclave; otherwise the page cannot be read 
from inside the enclave. 

WwW 1 If 1 indicates that the page can be written from inside the enclave; otherwise the page cannot be writ- 
ten from inside the enclave. 

X 2 If 1 indicates that the page can be executed from inside the enclave; otherwise the page cannot be 
executed from inside the enclave. 

PENDING 3 If 1 indicates that the page is in the PENDING state; otherwise the page is not in the PENDING state. 

MODIFIED 4 If 1 indicates that the page is in the MODIFIED state; otherwise the page is not in the MODIFIED state. 

PR If 1 indicates that a permission restriction operation on the page is in progress, otherwise a permission 
restriction operation is not in progress. 

RESERVED 7:6 Must be zero. 

PAGE_TYPE | 15:8 The type of page that the SECINFO is associated with. 

RESERVED 63:16 Must be zero. 


33.12.2 PAGE_TYPE Field Definition 
The SECINFO flags and EPC flags contain bits indicating the type of page. 


Table 33-19. Supported PAGE_TYPE 


TYPE Value Description 

PT_SECS 0 Page is an SECS. 

PT_TCS 1 Page is a TCS. 

PT_REG 2 Page is a regular page. 

PT_VA 3 Page is a Version Array. 

PT_TRIM 4 Page is in trimmed state. 

PT_SS_FIRST 5 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, Page is first page of a shadow stack. When 
CPUID.(EAX=1 2H, ECX=1):EAX[6] is 0, this value is reserved. 

PT_SS_REST 6 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, Page is not first page of a shadow stack. When 
CPUID.(EAX=1 2H, ECX=1):EAX[6] is 0, this value is reserved. 

All others Reserved. 


33.13 PAGING CRYPTO METADATA (PCMD) 


The PCMD structure is used to keep track of crypto meta-data associated with a paged-out page. Combined with 
PAGEINFO, it provides enough information for the processor to verify, decrypt, and reload a paged-out EPC page. 
The size of the PCMD structure (128 bytes) is architectural. 


EWB calculates the Message Authentication Code (MAC) value and writes out the PCMD. ELDB/U reads the fields 
and checks the MAC. 


The format of PCMD is as follows: 
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Table 33-20. Layout of PCMD Data Structure 


Field OFFSET (Bytes) | Size (Bytes) Description 
SECINFO 0 64 Flags describing the state of the enclave page; R/W by software. 
ENCLAVEID 64 8 Enclave Identifier used to establish a cryptographic binding between paged-out 
page and the enclave. 
RESERVED 72 40 Must be zero. 
MAC 112 16 Message Authentication Code for the page, page meta-data and reserved 
field. 


33.14 ENCLAVE SIGNATURE STRUCTURE (SIGSTRUCT) 


SIGSTRUCT is a structure created and signed by the enclave developer that contains information about the 
enclave. SIGSTRUCT is processed by the EINIT leaf function to verify that the enclave was properly built. 


SIGSTRUCT includes ENCLAVEHASH as SHA256 digest, as defined in FIPS PUB 180-4. The digests are byte strings 
of length 32. Each of the 8 HASH dwords is stored in little-endian order. 


SIGSTRUCT includes four 3072-bit integers (MODULUS, SIGNATURE, Q1, Q2). Each such integer is represented as 
a byte strings of length 384, with the most significant byte at the position “offset + 383”, and the least significant 
byte at position “offset”. 


The (3072-bit integer) SIGNATURE should be an RSA signature, where: a) the RSA modulus (MODULUS) is a 3072- 
bit integer; b) the public exponent is set to 3; c) the signing procedure uses the EMSA-PKCS1-v1.5 format with DER 
encoding of the “DigestInfo” value as specified in of PKCS#1 v2.1/RFC 3447. 


The 3072-bit integers Qi and Q2 are defined by: 

qi = floor(Signature*2 / Modulus); 

q2 = floor((Signature*3 - qi * Signature * Modulus) / Modulus); 
SIGSTRUCT must be page aligned 


In column 5 of Table 33-21, ‘Y’ indicates that this field should be included in the signature generated by the devel- 
oper. 


Table 33-21. Layout of Enclave Signature Structure (SIGSTRUCT) 


Field OFFSET (Bytes) | Size (Bytes) Description Signed 

HEADER 6) 16 Must be byte stream Y 
O6000000E1 000000000001 0000000000H 

VENDOR 16 4 Intel Enclave: OOOO8086H Y 
Non-Intel Enclave: OOOOOO000H 

DATE 20 4 Build date is yyyymmdd in hex: Y 
yyyy=4 digit year, mm=1-12, dd=1-31 

HEADER2 24 16 Must be byte stream Y 
01010000600000006000000001000000H 

SWDEFINED 40 4 Available for software use. Y 

RESERVED 44 84 Must be zero. Y 

MODULUS 128 384 Module Public Key (keylength=3072 bits). N 

EXPONENT 512 4 RSA Exponent = 3. N 

SIGNATURE 516 384 Signature over Header and Body. N 

MISCSELECT* 900 4 Bit vector specifying Extended SSA frame feature set tobe | Y 
used. 

MISCMASK* 904 4 Bit vector mask of MISCSELECT to enforce. Y 
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Table 33-21. Layout of Enclave Signature Structure (SIGSTRUCT) 


Field OFFSET (Bytes) | Size (Bytes) Description Signed 
CET_ATTRIBUTES | 908 1 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, this field pro- Y 
vides the Enclave CET attributes that must be set. When 
CPUID.(EAX=1 2H, ECX=1):EAX[6] is 0, this field is reserved 
and must be 0. 
CET_ATTRIBUTES | 909 1 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, this field pro- Y 
_MASK vides the Mask of CET attributes to enforce. When 
CPUID.(EAX=1 2H, ECX=1):EAX[6] is 0, this field is reserved 
and must be 0. 
RESERVED 910 2 Must be zero. Y 
ISVFAMILYID 912 16 ISV assigned Product Family ID. Y 
ATTRIBUTES 928 16 Enclave Attributes that must be set. Y 
ATTRIBUTEMASK | 944 16 Mask of Attributes to enforce. Y 
ENCLAVEHASH 960 32 MRENCLAVE of enclave this structure applies to. Y 
RESERVED 992 16 Must be zero. Y 
ISVEXTPRODID 1008 16 ISV assigned extended Product ID. Y 
ISVPRODID 1024 2 ISV assigned Product ID. Y 
ISVSVN 1026 2 ISV assigned SVN (security version number). Y 
RESERVED 1028 12 Must be zero. N 
Q1 1040 384 Q1 value for RSA Signature Verification. N 
Q2 1424 384 Q2 value for RSA Signature Verification. N 
* If CPUID.(EAX=12H, ECX=0):EBX[31:0] = 0, MISCSELECT must be 0. 
If CPUID.(EAX=12H, ECX=0):EBX[31:0] !=0, enclave writers must specify MISCSELECT such that each cleared 
bit in MISCMASK must also specify the corresponding bit as 0 in MISCSELECT. 


33.15 EINIT TOKEN STRUCTURE (EINITTOKEN) 


The EINIT token is used by EINIT to verify that the enclave is permitted to launch. EINIT token is generated by an 
enclave in possession of the EINITTOKEN key (the Launch Enclave). 


EINIT token must be 512-Byte aligned. 
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Table 33-22. Layout of EINIT Token (EINITTOKEN) 
Field OFFSET (Bytes) | Size (Bytes) | MACed Description 

Valid 0 4 Y Bit O: 1: Valid; O: Invalid. 
All other bits reserved. 

RESERVED 4 44 Y Must be zero. 

ATTRIBUTES 48 16 Y ATTRIBUTES of the Enclave. 

MRENCLAVE 64 32 Y MRENCLAVE of the Enclave. 

RESERVED 96 32 Y Reserved. 

MRSIGNER 128 32 Y MRSIGNER of the Enclave. 

RESERVED 160 32 Y Reserved. 

CPUSVNLE 192 16 N Launch Enclave’s CPUSVN. 

ISVPRODIDLE 208 02 N Launch Enclave'’s ISVPRODID. 

ISVSVNLE 210 02 N Launch Enclave's ISVSVN. 

CET_MASKED_AT | 212 1 N When CPUID.(EAX=1 2H, ECX=1):EAX[6] is 1, this field provides the 

TRIBUTES_LE Launch enclaves masked CET attributes. This should be set to LE’s 
CET_ATTRIBUTES masked with CET_ATTTRIBUTES_MASK of the LE's 
KEYREQUEST. When CPUID.(EAX=1 2H, ECX=1):EAX[6] is 0, this field is 
reserved. 

RESERVED 213 23 N Reserved. 

MASKEDMISCSEL | 236 4 Launch Enclave’s MASKEDMISCSELECT: set by the LE to the resolved 

ECTLE MISCSELECT value, used by EGETKEY (after applying KEYREQUEST's 
masking). 

MASKEDATTRIBU | 240 16 N Launch Enclave’s MASKEDATTRIBUTES: This should be set to the LE's 

TESLE ATTRIBUTES masked with ATTRIBUTEMASK of the LE’s KEYREQUEST. 

KEYID 256 32 N Value for key wear-out protection. 

MAC 288 16 N Message Authentication Code on EINITTOKEN using EINITTOKEN_KEY. 


33.16 REPORT (REPORT) 


The REPORT structure is the output of the EREPORT instruction, and must be 512-Byte aligned. 


Table 33-23. Layout of REPORT 


Field OFFSET (Bytes) | Size (Bytes) Description 

CPUSVN 0 16 The security version number of the processor. 

MISCSELECT 16 4 Bit vector specifying which extended features are saved to the MISC region of 
the SSA frame when an AEX occurs. 

CET_ATTRIBUTES | 20 1 When CPUID.(EAX=1 2H, ECX=1):EAX[6] is 1, this field reports the CET_ATTRIB- 
UTES of the Enclave. When CPUID.(EAX=12H, ECX=1):EAX[6] is 0, this field is re- 
served and must be 0. 

RESERVED 20 12 Zero. 

ISVEXTNPRODID | 32 16 The value of SECS.ISVEXTPRODID. 

ATTRIBUTES 48 16 ATTRIBUTES of the Enclave. See Section 33.7.1. 

MRENCLAVE 64 32 The value of SECS.MRENCLAVE. 

RESERVED 96 32 Zero. 

MRSIGNER 128 32 The value of SECS.MRSIGNER. 

RESERVED 160 32 Zero. 
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Table 33-23. Layout of REPORT 


Field OFFSET (Bytes) | Size (Bytes) Description 
CONFIGID 192 64 Value provided by SW to identify enclave’s post EINIT configuration. 
ISVPRODID 256 2 Product ID of enclave. 
ISVSVN 258 2 Security version number (SVN) of the enclave. 
CONFIGSVN 260 2 Value provided by SW to indicate expected SVN of enclave’s post EINIT configu- 
ration. 
RESERVED 262 42 Zero. 
ISVFAMILYID 304 16 The value of SECS.ISVFAMILYID. 
REPORTDATA 320 64 Data provided by the user and protected by the REPORT's MAC, see Section 
33.16.1. 
KEYID 384 32 Value for key wear-out protection. 
MAC 416 16 Message Authentication Code on the report using report key. 
33.16.1 REPORTDATA 


REPORTDATA is a 64-Byte data structure that is provided by the enclave and included in the REPORT. It can be used 
to securely pass information from the enclave to the target enclave. 


33.17 REPORT TARGET INFO (TARGETINFO) 


This structure is an input parameter to the EREPORT leaf function. The address of TARGETINFO is specified as an 
effective address in RBX. It is used to identify the target enclave which will be able to cryptographically verify the 
REPORT structure returned by EREPORT. TARGETINFO must be 512-Byte aligned. 


Table 33-24. Layout of TARGETINFO Data Structure 


Field OFFSET (Bytes) | Size (Bytes) Description 

MEASUREMENT |0 32 The MRENCLAVE of the target enclave. 

ATTRIBUTES 32 16 The ATTRIBUTES field of the target enclave. 

CET_ATTRIBUTES | 48 1 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, this field provides the 
CET_ATTRIBUTES field of the target enclave. When CPUID.(EAX=1 2H, 
ECX=1):EAX[6] is 0, this field is reserved. 

RESERVED 49 1 Must be zero. 

CONFIGSVN 50 2 CONFIGSVN of the target enclave. 

MISCSELECT 52 4 The MISCSELECT of the target enclave. 

RESERVED 56 8 Must be zero. 

CONFIGID 64 64 COMFIGID of target enclave. 

RESERVED 128 384 Must be zero. 


33.18 KEY REQUEST (KEYREQUEST) 


This structure is an input parameter to the EGETKEY leaf function. It is passed in as an effective address in RBX and 
must be 512-Byte aligned. It is used for selecting the appropriate key and any additional parameters required in 
the derivation of that key. 
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Table 33-25. Layout of KEYREQUEST Data Structure 


Field OFFSET (Bytes) | Size (Bytes) Description 
KEYNAME 0 2 Identifies the Key Required. 
KEYPOLICY 2 2 Identifies which inputs are required to be used in the key derivation. 
ISVSVN 4 2 The ISV security version number that will be used in the key derivation. 
CET_ATTRIBUTES | 6 1 When CPUID.(EAX=12H, ECX=1):EAX[6] is 1, this field provides a mask that 
_MASK defines which CET_ATTRIBUTES bits will be included in key derivation. When 
CPUID.(EAX=12H, ECX=1):EAX[6] is 0, then this field is reserved and must be 
0. 
RESERVED 7 1 Must be zero. 
CPUSVN 8 16 The security version number of the processor used in the key derivation. 
ATTRIBUTEMASK | 24 16 A mask defining which ATTRIBUTES bits will be included in key derivation. 
KEYID 40 32 Value for key wear-out protection. 
MISCMASK 72 4 A mask defining which MISCSELECT bits will be included in key derivation. 
CONFIGSVN 76 2 Identifies which enclave Configuration’s Security Version should be used in key 
derivation. 
RESERVED 78 434 
33.18.1 KEY REQUEST KeyNames 
Table 33-26. Supported KEYName Values 
Key Name Value Description 


EINITTOKEN_KEY 


EINIT_TOKEN key 


PROVISION_KEY 


Provisioning Key 


Provisioning Seal Key 


0 
1 
PROVISION_SEAL_KEY | 2 
3 
4 


REPORT_KEY Report Key 
SEAL_KEY Seal Key 
All others Reserved 


33.18.2 Key Request Policy Structure 


Table 33-27. Layout of KEYPOLICY Field 


Field Bit Position Description 
MRENCLAVE 0 If 1, derive key using the enclave’s MRENCLAVE measurement register. 
MRSIGNER 1 If 1, derive key using the enclave’s MRSIGNER measurement register. 
NOISVPRODID 2 If 1, derive key WITHOUT using the enclave’ ISVPRODID value. 
CONFIGID 3 If 1, derive key using the enclave’s CONFIGID value. 
ISVFAMILYID 4 If 1, derive key using the enclave ISVFAMILYID value. 
ISVEXTPRODID 5 If 1, derive key using enclave’s ISVEXTPRODID value. 
RESERVED 15:6 Must be zero. 
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33.19 VERSION ARRAY (VA) 


In order to securely store the versions of evicted EPC pages, Intel SGX defines a special EPC page type called a 
Version Array (VA). Each VA page contains 512 slots, each of which can contain an 8-byte version number for a 
page evicted from the EPC. When an EPC page is evicted, software chooses an empty slot in a VA page; this slot 
receives the unique version number of the page being evicted. When the EPC page is reloaded, there must be a VA 
slot that must hold the version of the page. If the page is successfully reloaded, the version in the VA slot is 
cleared. 


VA pages can be evicted, just like any other EPC page. When evicting a VA page, a version slot in some other VA 
page must be used to hold the version for the VA being evicted. A Version Array Page must be 4K-Bytes aligned. 


Table 33-28. Layout of Version Array Data Structure 


Field OFFSET (Bytes) | Size (Bytes) Description 
Slot 0 0 8 Version Slot 0 
Slot 1 8 8 Version Slot 1 
Slot 511 4088 8 Version Slot 511 


33.20 ENCLAVE PAGE CACHE MAP (EPCM) 


EPCM is a secure structure used by the processor to track the contents of the EPC. The EPCM holds exactly one 
entry for each page that is currently loaded into the EPC. EPCM is not accessible by software, and the layout of 
EPCM fields is implementation specific. 


Table 33-29. Content of an Enclave Page Cache Map Entry 


Field Description 
VALID Indicates whether the EPCM entry is valid. 
(R ~~___ | Read access; indicates whether enclave accesses for reads are allowed from the EPC page referenced by this | 

entry. 

W Write access; indicates whether enclave accesses for writes are allowed to the EPC page referenced by this 
entry. 

X Execute access; indicates whether enclave accesses for instruction fetches are allowed from the EPC page 
referenced by this entry. 

PT EPCM page type (PT_SECS, PT_TCS, PT_REG, PT_VA, PT_TRIM, PT_SS_FIRST, PT_SS_ REST). 

ENCLAVESECS SECS identifier of the enclave to which the EPC page belongs. 

ENCLAVEADDRESS Linear enclave address of the EPC page. 

BLOCKED Indicates whether the EPC page is in the blocked state. 

PENDING Indicates whether the EPC page is in the pending state. 

MODIFIED Indicates whether the EPC page is in the modified state. 

PR Indicates whether the EPC page is in a permission restriction state. 


33.21 READ INFO (RDINFO) 


The RDINFO structure contains status information about an EPC page. It must be aligned to 32-Bytes. 
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Table 33-30. Layout of RDINFO Structure 


Field nite a Size (Bytes) Description 
STATUS 0 8 Page status information. 
FLAGS 8 EPCM state of the page. 
ENCLAVECONTEXT 16 8 Context pointer describing the page's parent location. 
33.21.1 RDINFO Status Structure 
Table 33-31. Layout of RDINFO STATUS Structure 
Field Bit Position Description 
CHILDPRESENT 0) Indicates that the page has one or more child pages present (always zero for non-SECS 
pages). In VMX non-root operation includes the presence of virtual children. 
VIRTCHLDPRESENT 1 Indicates that the page has one or more virtual child pages present (always zero for non- 
SECS pages). In VMX non-root operation this value is always zero. 
RESERVED 63:2 


33.21.2 RDINFO Flags Structure 


Table 33-32. Layout of RDINFO FLAGS Structure 


Field Bit Position Description 

R 0 Read access; indicates whether enclave accesses for reads are allowed from the EPC page 
referenced by this entry. 

W 1 Write access; indicates whether enclave accesses for writes are allowed to the EPC page 
referenced by this entry. 

X 2 Execute access; indicates whether enclave accesses for instruction fetches are allowed 
from the EPC page referenced by this entry. 

PENDING 3 Indicates whether the EPC page is in the pending state. 

MODIFIED 4 Indicates whether the EPC page is in the modified state. 

PR 5 Indicates whether the EPC page is in a permission restriction state. 

RESERVED 7:6 

PAGE_TYPE 15:8 Indicates the page type of the EPC page. 

RESERVED 62:16 

BLOCKED 63 Indicates whether the EPC page is in the blocked state. 
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CHAPTER 34 
ENCLAVE OPERATION 


The following aspects of enclave operation are described in this chapter: 


¢ Enclave creation: Includes loading code and data from outside of enclave into the EPC and establishing the 
enclave entity. 


¢ Adding pages and measuring the enclave. 


Initialization of an enclave: Finalizes the cryptographic log and establishes the enclave identity and sealing 
identity. 


¢ Enclave entry and exiting including: 
— Controlled entry and exit. 


— Asynchronous Enclave Exit (AEX) and resuming execution after an AEX. 


34.1 CONSTRUCTING AN ENCLAVE 


Figure 34-1 illustrates a typical Enclave memory layout. 


SECS ee ee 
~ QOS Context 
Base + Size ———> + 
Enclave Memory 
a Thread Data 
Replicated once — 
per thread ~ — TCs 
. _ - Application Context 
le 
Global Data = 
Code 
Base __ 5 /) 
Enclave {Base, Size} 


Figure 34-1. Enclave Memory Layout 


The enclave creation, commitment of memory resources, and finalizing the enclave’s identity with measurement 
comprises multiple phases. This process can be illustrated by the following exemplary steps: 


1. The application hands over the enclave content along with additional information required by the enclave 
creation API to the enclave creation service running at privilege level 0. 


2. The enclave creation service running at privilege level 0 uses the ECREATE leaf function to set up the initial 
environment, specifying base address and size of the enclave. This address range, the ELRANGE, is part of the 
application's address space. This reserves the memory range. The enclave will now reside in this address 
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region. ECREATE also allocates an Enclave Page Cache (EPC) page for the SGX Enclave Control Structure 
(SECS). Note that this page is not required to be a part of the enclave linear address space and is not required 
to be mapped into the process. 


3. The enclave creation service uses the EADD leaf function to commit EPC pages to the enclave, and use 
EEXTEND to measure the committed memory content of the enclave. For each page to be added to the enclave: 


— Use EADD to add the new page to the enclave. 


— Ifthe enclave developer requires measurement of the page as a proof for the content, use EEXTEND to add 
a measurement for 256 bytes of the page. Repeat this operation until the entire page is measured. 


4. The enclave creation service uses the EINIT leaf function to complete the enclave creation process and finalize 
the enclave measurement to establish the enclave identity. Until an EINIT is executed, the enclave is not 
permitted to execute any enclave code (i.e. entering the enclave by executing EENTER would result in a fault). 


34.1.1 ECREATE 


The ECREATE leaf function sets up the initial environment for the enclave by reading an SGX Enclave Control Struc- 
ture (SECS) that contains the enclave's address range (ELRANGE) as defined by BASEADDR and SIZE, the ATTRI- 
BUTES and MISCSELECT bitmaps, and the SSAFRAMESIZE. It then securely stores this information in an Enclave 
Page Cache (EPC) page. ELRANGE is part of the application's address space. ECREATE also initializes a crypto- 
graphic log of the enclave's build process. 


34.1.2 EADD and EEXTEND Interaction 


Once the SECS has been created, enclave pages can be added to the enclave via EADD. This involves converting a 
free EPC page into either a PT_REG or a PT_TCS page. 


When EADD is invoked, the processor will update the EPCM entry with the type of page (PT_REG or PT_TCS), the 
linear address used by the enclave to access the page, and the enclave access permissions for the page. It associ- 
ates the page to the SECS provided as input. The EPCM entry information is used by hardware to manage access 
control to the page. EADD records EPCM information in the cryptographic log stored in the SECS and copies 4 
KBytes of data from unprotected memory outside the EPC to the allocated EPC page. 


System software is responsible for selecting a free EPC page. System software is also responsible for providing the 
type of page to be added, the attributes of the page, the contents of the page, and the SECS (enclave) to which the 
page is to be added as requested by the application. Incorrect data would lead to a failure of EADD or to an incor- 
rect cryptographic log and a failure at EINIT time. 


After a page has been added to an enclave, software can measure a 256 byte region as determined by the devel- 
oper by invoking EEXTEND. Thus to measure an entire 4KB page, system software must execute EEXTEND 16 
times. Each invocation of EEXTEND adds to the cryptographic log information about which region is being measured 
and the measurement of the section. 


Entries in the cryptographic log define the measurement of the enclave and are critical in gaining assurance that 
the enclave was correctly constructed by the untrusted system software. 


34.1.3 EINIT Interaction 


Once system software has completed the process of adding and measuring pages, the enclave needs to be initial- 
ized by the EINIT leaf function. After an enclave is initialized, EADD and EEXTEND are disabled for that enclave (An 
attempt to execute EADD/EEXTEND to enclave after enclave initialization will result in a fault). The initialization 
process finalizes the cryptographic log and establishes the enclave identity and sealing identity used by 
EGETKEY and EREPORT. 


A cryptographic hash of the log is stored as the enclave identity. Correct construction of the enclave results in the 
cryptographic hash matching the one built by the enclave owner and included as the ENCLAVEHASH field of 
SIGSTRUCT. The enclave identity provided by the EREPORT leaf function can be verified by a remote party. 
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The EINIT leaf function checks the EINIT token to validate that the enclave has been enabled on this platform. If 
the enclave is not correctly constructed, or the EINIT token is not valid for the platform, or SIGSTRUCT isn't prop- 
erly signed, then EINIT will fail. See the EINIT leaf function for details on the error reporting. 


The enclave identity is a cryptographic hash that reflects the enclave attributes and MISCSELECT value, content 
of the enclave, the order in which it was built, the addresses it occupies in memory, the security attributes, and 
access right permissions of each page. The enclave identity is established by the EINIT leaf function. 


The sealing identity is managed by a sealing authority represented by the hash of the public key used to sign the 
SIGSTRUCT structure processed by EINIT. The sealing authority assigns a product ID (ISVPRODID) and security 
version number (ISVSVN) to a particular enclave identity. 

EINIT establishes the sealing identity using the following steps: 

1. Verifies that SIGSTRUCT is properly signed using the public key enclosed in the SIGSTRUCT. 

2. Checks that the measurement of the enclave matches the measurement of the enclave specified in SIGSTRUCT. 
3. Checks that the enclave’s attributes and MISCSELECT values are compatible with those specified in SIGSTRUCT. 


4. Finalizes the measurement of the enclave and records the sealing identity (the sealing authority, product id 
and security version number) and enclave identity in the SECS. 


5. Sets the ATTRIBUTES. INIT bit for the enclave. 


34.1.4 — Intel® SGX Launch Control Configuration 


Intel® SGX Launch Control is a set of controls that govern the creation of enclaves. Before the EINIT leaf function 
will successfully initialize an enclave, a designated Launch Enclave must create an EINITTOKEN for that enclave. 
Launch Enclaves have SECS.ATTRIBUTES.EINITTOKEN_KEY = 1, granting them access to the EINITTOKEN_KEY 
from the EGETKEY leaf function. EINITTOKEN_KEY must be used by the Launch Enclave when computing EINIT- 
TOKEN.MAC, the Message Authentication Code of the EINITTOKEN. 


The hash of the public key used to sign the SIGSTRUCT of the Launch Enclave must equal the value in the 
IA32_SGXLEPUBKEYHASH MSRs. Only Launch Enclaves are allowed to launch without a valid token. 


The IA32_SGXLEPUBKEYHASH MSRs are provided to designate the platform’s Launch Enclave. 
IA32_SGXLEPUBKEYHASH defaults to digest of Intel’s launch enclave signing key after reset. 


IA32_FEATURE_CONTROL bit 17 controls the permissions on the IA32_SGXLEPUBKEYHASH MSRs when 
CPUID.(EAX=12H, ECX=00H):EAX[0] = 1. If IA32_FEATURE_CONTROL is locked with bit 17 set, 
IA32_SGXLEPUBKEYHASH MSRs are reconfigurable (writeable). If either IA32_FEATURE_CONTROL is not locked or 
bit 17 is clear, the MSRs are read only. By leaving these MSRs writable, system SW or a VMM can support a plurality 
of Launch Enclaves for hosting multiple execution environments. See Table 38.2.2 for more details. 


34.2 ENCLAVE ENTRY AND EXITING 


34.2.1. Controlled Entry and Exit 


The EENTER leaf function is the method to enter the enclave under program control. To execute EENTER, software 
must supply an address of a TCS that is part of the enclave to be entered. The TCS holds the location inside the 
enclave to transfer control to and a pointer to the SSA frame inside the enclave that an AEX should store the 
register state to. 


When a logical processor enters an enclave, the TCS is considered busy until the logical processors exits the 
enclave. An attempt to enter an enclave through a busy TCS results in a fault. Intel® SGX allows an enclave builder 
to define multiple TCSs, thereby providing support for multithreaded enclaves. 


Software must also supply to EENTER the Asynchronous Exit Pointer (AEP) parameter. AEP is an address external 
to the enclave which an exception handler will return to using IRET. Typically the location would contain the 
ERESUME instruction. ERESUME transfers control back to the enclave, to the address retrieved from the enclave 
thread’s saved state. 


EENTER performs the following operations: 
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Check that TCS is not busy and flush all cached linear-to-physical mappings. 
Change the mode of operation to be in enclave mode. 


Save the old RSP, RBP for later restore on AEX (Software is responsible for setting up the new RSP, RBP to be 
used inside enclave). 


Save XCRO and replace it with the XFRM value for the enclave. 
5. Check if software wishes to debug (applicable to a debuggable enclave): 

— If not debugging, then configure hardware so the enclave appears as a single instruction. 

— If debugging, then configure hardware to allow traps, breakpoints, and single steps inside the enclave. 
6. Set the TCS as busy. 
7. Transfer control from outside enclave to predetermined location inside the enclave specified by the TCS. 


The EEXIT leaf function is the method of leaving the enclave under program control. EEXIT receives the target 
address outside of the enclave that the enclave wishes to transfer control to. It is the responsibility of enclave soft- 
ware to erase any secret from the registers prior to invoking EEXIT. To allow enclave software to easily perform an 
external function call and re-enter the enclave (using EEXIT and EENTER leaf functions), EEXIT returns the value of 
the AEP that was used when the enclave was entered. 


EEXIT performs the following operations: 
1. Clear enclave mode and flush all cached linear-to-physical mappings. 
2. Mark TCS as not busy. 


3. Transfer control from inside the enclave to a location on the outside specified as parameter to the EEXIT leaf 
function. 


34.2.2 Asynchronous Enclave Exit (AEX) 


Asynchronous and synchronous events, such as exceptions, interrupts, traps, SMIs, and VM exits may occur while 
executing inside an enclave. These events are referred to as Enclave Exiting Events (EEE). Upon an EEE, the 
processor state is securely saved inside the enclave (in the thread’s current SSA frame) and then replaced by a 
synthetic state to prevent leakage of secrets. The process of securely saving state and establishing the synthetic 
state is called an Asynchronous Enclave Exit (AEX). Details of AEX is described in Chapter 35, “Enclave Exiting 
Events”. 


As part of most EEEs, the AEP is pushed onto the stack as the location of the eventing address. This is the location 
where control will return to after executing the IRET. The ERESUME leaf function can be executed from that point 
to reenter the enclave and resume execution from the interrupted point. 


After AEX has completed, the logical processor is no longer in enclave mode and the exiting event is processed 
normally. Any new events that occur after the AEX has completed are treated as having occurred outside the 
enclave (e.g. a #PF in dispatching to an interrupt handler). 


34.2.3 Resuming Execution after AEX 


After system software has serviced the event that caused the logical processor to exit an enclave, the logical 
processor can continue enclave execution using ERESUME. ERESUME restores processor state and returns control 
to where execution was interrupted. 


If the cause of the exit was an exception or a fault and was not resolved, the event will be triggered again if the 
enclave is re-entered using ERESUME. For example, if an enclave performs a divide by 0 operation, executing 
ERESUME will cause the enclave to attempt to re-execute the faulting instruction and result in another divide by 0 
exception. Intel® SGX provides the means for an enclave developer to handle enclave exceptions from within the 
enclave. Software can enter the enclave at a different location and invoke the exception handler within the enclave 
by executing the EENTER leaf function. The exception handler within the enclave can read the fault information 
from the SSA frame and attempt to resolve the faulting condition or simply return and indicate to software that the 
enclave should be terminated (e.g. using EEXIT). 
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34.2.3.1 ERESUME Interaction 
ERESUME restores registers depending on the mode of the enclave (32 or 64 bit). 


¢° In 32-bit mode (IA32_EFER.LMA = 0 || CS.L = 0), the low 32-bits of the legacy registers (EAX, EBX, ECX, EDX, 
ESP, EBP, ESI, EDI, EIP and EFLAGS) are restored from the thread’s GPR area of the current SSA frame. Neither 
the upper 32 bits of the legacy registers nor the 64-bit registers (R8 ...R15) are loaded. 


° In 64-bit mode (IA32_EFER.LMA = 1 && CS.L = 1), all 64 bits of the general processor registers (RAX, RBX, 
RCX, RDX, RSP, RBP, RSI, RDI, R8 ... R15, RIP and RFLAGS) are loaded. 


Extended features specified by SECS.ATTRIBUTES.XFRM are restored from the XSAVE area of the current SSA 
frame. The layout of the x87 area depends on the current values of IA32_EFER.LMA and CS.L: 


¢ JTA32_EFER.LMA = 0 || CS.L=0 
— 32-bit load in the same format that XSAVE/FXSAVE uses with these values. 
¢ = JA32_EFER.LMA = 1 &&CS.L = 1 
— 64-bit load in the same format that XSAVE/FXSAVE uses with these values as if REX.W = 1. 


34.3. CALLING ENCLAVE PROCEDURES 


34.3.1 Calling Convention 


In standard call conventions subroutine parameters are generally pushed onto the stack. The called routine, being 
aware of its own stack layout, knows how to find parameters based on compile-time-computable offsets from the 
SP or BP register (depending on runtime conventions used by the compiler). 


Because of the stack switch when calling an enclave, stack-located parameters cannot be found in this manner. 
Entering the enclave requires a modified parameter passing convention. 


For example, the caller might push parameters onto the untrusted stack and then pass a pointer to those parame- 
ters in RAX to the enclave software. The exact choice of calling conventions is up to the writer of the edge routines; 
be those routines hand-coded or compiler generated. 


34.3.2 Register Preservation 


As with most systems, it is the responsibility of the callee to preserve all registers except that used for returning a 
value. This is consistent with conventional usage and tends to optimize the number of register save/restore oper- 
ations that need be performed. It has the additional security result that it ensures that data is scrubbed from any 
registers that were used by enclave to temporarily contain secrets. 


34.3.3. Returning to Caller 


No registers are modified during EEXIT. It is the responsibility of software to remove secrets in registers before 
executing EEXIT. 


34.4 INTEL® SGX KEY AND ATTESTATION 


34.4.1 Enclave Measurement and Identification 


During the enclave build process, two “measurements” are taken of each enclave and are stored in two 256-bit 
Measurement Registers (MR): MRENCLAVE and MRSIGNER. MRENCLAVE represents the enclave's contents and 
build process. MRSIGNER represents the entity that signed the enclave's SIGSTRUCT. 
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The values of the Measurement Registers are included in attestations to identify the enclave to remote parties. The 
MRs are also included in most keys, binding keys to enclaves with specific MRs. 


34.4.1.1 MRENCLAVE 


MRENCLAVE is a unique 256 bit value that identifies the code and data that was loaded into the enclave during the 
initial launch. It is computed as a SHA256 hash that is initialized by the ECREATE leaf function. EADD and EEXTEND 
leaf functions record information about each page and the content of those pages. The EINIT leaf function finalizes 
the hash, which is stored in SECS.MRENCLAVE. Any tampering with the build process, contents of a page, page 
permissions, etc will result in a different MRENCLAVE value. 


Figure 34-2 illustrates a simplified flow of changes to the MRENCLAVE register when building an enclave: 
¢ Enclave creation with ECREATE. 

® Copying a non-enclave source page into the EPC of an un-initialized enclave with EADD. 

¢ Updating twice of the MRENCLAVE after modifying the enclave’s page content, i.e. EEXTEND twice. 

¢ Finalizing the enclave build with EINIT. 

Details on specific values inserted in the hash are available in the individual instruction definitions. 


Page Data Chunk 1 Data Chunk 2 
Metadata Chunk 1 Metadata Chunk 2 Metadata 


y y y 
SHA_INIT r= SHA_UPDATE ;— SHA_UPDATE > SHA_UPDATE => SHA FINAL 


| Ml y y | 
[ uRENCLAVE [ uRENCLAVE MRENCLAVE MRENCLAVE i MRENCLAVE 


ECREATE EADD EEXTEND EEXTEND EINIT 


Figure 34-2. Measurement Flow of Enclave Build Process 


34.4.1.2 MRSIGNER 


Each enclave is signed using a 3072 bit RSA key. The signature is stored in the SIGSTRUCT. In the SIGSTRUCT, the 
enclave's signer also assigns a product ID (ISVPRODID) and a security version (ISVSVN) to the enclave. 
MRSIGNER is the SHA-256 hash of the signer's public key. For platforms that support Key Separation and Sharing 
(CPUID.(EAX=12H, ECX=1).EAX.KSS[7]) the SIGSTRUCT can additionally specify an 16 byte extended product ID 
(ISVEXTPRODID), and a 16 byte family ID (ISVFAMILYID). 


In attestation, MRSIGNER can be used to allow software to approve of an enclave based on the author rather than 
maintaining a list of MRENCLAVEs. It is used in key derivation to allow software to create a lineage of an applica- 
tion. By signing multiple enclaves with the same key, the enclaves will share the same keys and data. Combined 
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with security version numbering, the author can release multiple versions of an application which can access keys 
for previous versions, but not future versions of that application. 


34.4.1.3 CONFIGID 


For platforms that support enhancements for key separation and sharing (CPUID.(EAX=12H, ECX=1).EAX.KSS[7]) 
when the enclave is created the platform can additionally provide 32-byte configuration identifier (CONFIGID). 
How this value is used is dependent on the enclave but it is intended to allow enclave creators to indicate what 
additional content may be accepted by the enclave post-initialization. 


34.4.2 Security Version Numbers (SVN) 


Intel® SGX supports a versioning system that allows the signer to identify different versions of the same software 
released by an author. The security version is independent of the functional version an author uses and is intended 
to specify security equivalence. Multiple releases with functional enhancements may all share the same SVN if they 
all have the same security properties or posture. Each enclave has an SVN and the underlying hardware has an 
SVN. 


The SVNs are attested to in EREPORT and are included in the derivation of most keys, thus providing separation 
between data for older/newer versions. 


34.4.2.1 Enclave Security Version 


In the SIGSTRUCT, the MRSIGNER is associated with a 16-bit Product ID (ISVPRODID) and a 16 bit integer SVN 
(ISVSVN). Together they define a specific group of versions of a specific product. Most keys, including the Seal Key, 
can be bound to this pair. 


To support upgrading from one release to another, EGETKEY will return keys corresponding to any value less than 
or equal to the software's ISVSVN. 


34.4.2.2 Hardware Security Version 


CPUSVN is a 128 bit value that reflects the microcode update version and authenticated code modules supported 
by the processor. Unlike ISVSVN, CPUSVN is not an integer and cannot be compared mathematically. Not all values 
are valid CPUSVNs. 


Software must ensure that the CPUSVN provided to EGETKEY is valid. EREPORT will return the CPUSVN of the 
current environment. Software can execute EREPORT with TARGETINFO set to zeros to retrieve a CPUSVN from 
REPORTDATA. Software can access keys for a CPUSVN recorded previously, provided that each of the elements 
reflected in CPUSVN are the same or have been upgraded. 


34.4.2.3 CONFIGID Security Version 


The CONFIGID field can be used to contain the hash of a signing key for verifying the additional content. In this 
case, similar to the relationship between MRSIGNER and ISVSVN, CONFIGID needs a CONFIGID Security Version 
Number. CONFIGIDSVN can be specified at the same time as CONFIGID. 


34.4.3. Keys 


Intel® SGX provides software with access to keys unique to each processor and rooted in HW keys inserted into the 
processor during manufacturing. 


Each enclave requests keys using the EGETKEY leaf function. The key is based on enclave parameters such as 
measurement, the enclave signing key, security attributes of the enclave, and the Hardware Security version of the 
processor itself. A full list of parameter options is specified in the KEYREQUEST structure, see details in Section 
33.18. 


By deriving keys using enclave properties, SGX guarantees that if two enclaves call EGETKEY, they will receive a 
unique key only accessible by the respective enclave. It also guarantees that the enclave will receive the same key 
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on every future execution of EGETKEY. Some parameters are optional or configurable by software. For example, a 

Seal key can be based on the signer of the enclave, resulting in a key available to multiple enclaves signed by the 

same party. 

The EGETKEY leaf function provides several key types. Each key is specific to the processor, CPUSVN, and the 

enclave that executed EGETKEY. The EGETKEY instruction definition details how each of these keys is derived, see 

Table 36-64. Additionally, 

¢ SEAL Key: The Seal key is a general purpose key for the enclave to use to protect secrets. Typical uses of the 
Seal key are encrypting and calculating MAC of secrets on disk. There are 2 types of Seal Key described in 
Section 34.4.3.1. 

¢ REPORT Key: This key is used to compute the MAC on the REPORT structure. The EREPORT leaf function is used 
to compute this MAC, and destination enclave uses the Report key to verify the MAC. The software usage flow 
is detailed in Section 34.4.3.2. 

¢ EINITTOKEN_KEY: This key is used by Launch Enclaves to compute the MAC on EINITTOKENs. These tokens are 
then verified in the EINIT leaf function. The key is only available to enclaves with ATTRIBUTE. EINITTOKEN_KEY set 
to 1. 

® PROVISIONING Key and PROVISIONING SEAL Key: These keys are used by attestation key provisioning 


software to prove to remote parties that the processor is genuine and identify the currently executing TCB. 
These keys are only available to enclaves with ATTRIBUTE.PROVISIONKEY set to 1. 


34.4.3.1. Sealing Enclave Data 


Enclaves can protect persistent data using Seal keys to provide encryption and/or integrity protection. EGETKEY 
provides two types of Seal keys specified in KEYREQUEST.KEYPOLICY field: MRENCLAVE-based key and 
MRSIGNER-based key. 


The MRENCLAVE-based keys are available only to enclave instances sharing the same MRENCLAVE. If a new 
version of the enclave is released, the Seal keys will be different. Retrieving previous data requires additional soft- 
ware support. 


The MRSIGNER-based keys are bound to the 3 tuple (MRSIGNER, ISVPRODID, ISVSVN). These keys are available 
to any enclave with the same MRSIGNER and ISVPRODID and an ISVSVN equal to or greater than the key in ques- 
tions. This is valuable for allowing new versions of the same software to retrieve keys created before an upgrade. 


For platforms that support enhancements for key separation and sharing (CPUID.(EAX=12H, ECX=1).EAX.KSS[7]) 
four additional key policies for seal key derivation are provided. These add the ISVEXTPRODID, ISVFAMILYID and 

CONFIGID/CONFIGSVN to the key derivation. Additionally there is a policy to remove ISVPRODID from a key deri- 
vation to create a shared between different products that share the same MRSIGNER. 


34.4.3.2 Using REPORTs for Local Attestation 


SGX provides a means for enclaves to securely identify one another, this is referred to as “Local Attestation”. SGX 
provides a hardware assertion, REPORT that contains calling enclaves Attributes, Measurements and User supplied 
data (described in detail in Section 33.16). Figure 34-3 shows the basic flow of information. 


1. The source enclave determines the identity of the target enclave to populate TARGETINFO. 


2. The source enclave calls EREPORT instruction to generate a REPORT structure. The EREPORT instruction 
conducts the following: 


— Populates the REPORT with identify information about the calling enclave. 


— Derives the Report Key that is returned when the target enclave executes the EGETKEY. TARGETINFO 
provides information about the target. 


— Computes a MAC over the REPORT using derived target enclave Report Key. 
3. Non-enclave software copies the REPORT from source to destination. 


The target enclave executes the EGETKEY instruction to request its REPORT key, which is the same key used by 
EREPORT at the source. 


5. The target enclave verifies the MAC and can then inspect the REPORT to identify the source. 
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Legend: REPORT 


Figure 34-3. SGX Local Attestation 


34.5  EPCAND MANAGEMENT OF EPC PAGES 


EPC layout is implementation specific, and is enumerated through CPUID (see Table 32-7 for EPC layout). EPC is 
typically configured by BIOS at system boot time. 


34.5.1 EPC Implementation 


EPC must be properly protected against attacks. One example of EPC implementation could use a Memory Encryp- 
tion Engine (MEE). An MEE provides a cost-effective mechanism of creating cryptographically protected volatile 
storage using platform DRAM. These units provide integrity, replay, and confidentiality protection. Details are 
implementation specific. 


34.5.2 OS Management of EPC Pages 


The EPC is a finite resource. SGX1 (i.e. CPUID.(EAX=12H, ECX=0):EAX.SGX1 = 1 but CPUID.(EAX=12H, 
ECX=0):EAX.SGX2 = 0) provides the EPC manager with leaf functions to manage this resource and properly swap 
pages out of and into the EPC. For that, the EPC manager would need to keep track of all EPC entries, type and 
state, context affiliation, and SECS affiliation. 


Enclave pages that are candidates for eviction should be moved to BLOCKED state using EBLOCK instruction that 
ensures no new cached virtual to physical address mappings can be created by attempts to reference a BLOCKED 
page. 

Before evicting blocked pages, EPC manager should execute ETRACK leaf function on that enclave and ensure that 
there are no stale cached virtual to physical address mappings for the blocked pages remain on any thread on the 
platform. 


After removing all stale translations from blocked pages, system software should use the EWB leaf function for 
securely evicting pages out of the EPC. EWB encrypts a page in the EPC, writes it to unprotected memory, and 
invalidates the copy in EPC. In addition, EWB also creates a cryptographic MAC (PCMD.MAC) of the page and stores 
it in unprotected memory. A page can be reloaded back to the processor only if the data and MAC match. To ensure 
that only the latest version of the evicted page can be loaded back, the version of the evicted page is stored 
securely in a Version Array (VA) in EPC. 


SGX1 includes two instructions for reloading pages that have been evicted by system software: ELDU and ELDB. 
The difference between the two instructions is the value of the paging state at the end of the instruction. ELDU 
results in a page being reloaded and set to an UNBLOCKED state, while ELDB results in a page loaded to a 
BLOCKED state. 
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ELDB is intended for use by a Virtual Machine Monitor (VMM). When a VMM reloads an evicted page, it needs to 
restore it to the correct state of the page (BLOCKED vs. UNBLOCKED) as it existed at the time the page was 
evicted. Based on the state of the page at eviction, the VMM chooses either ELDB or ELDU. 


34.5.2.1_ Enhancement to Managing EPC Pages 


On processors supporting SGX2 (i.e. CPUID.(EAX=12H, ECX=0):EAX.SGX2 = 1), the EPC manager can manage 
EPC resources (while enclave is running) with more flexibility provided by the SGX2 leaf functions. The additional 
flexibility is described in Section 34.5.7 through Section 34.5.11. 


34.5.3. Eviction of Enclave Pages 
Intel SGX paging is optimized to allow the Operating System (OS) to evict multiple pages out of the EPC under a 
single synchronization. 
The suggested flow for evicting a list of pages from the EPC is: 
1. For each page to be evicted from the EPC: 
a. Select an empty slot in a Version Array (VA) page. 
¢ If no empty VA page slots exist, create a new VA page using the EPA leaf function. 


b. Remove linear-address to physical-address mapping from the enclave context’s mapping tables (page table 
and EPT tables). 


c. Execute the EBLOCK leaf function for the target page. This sets the target page state to BLOCKED. At this 
point no new mappings of the page will be created. So any access which does not have the mapping cached 
in the TLB will generate a #PF. 


2. For each enclave containing pages selected in step 1: 


— Execute an ETRACK leaf function pointing to that enclave’s SECS. This initiates the tracking process that 
ensures that all caching of linear-address to physical-address translations for the blocked pages is cleared. 


3. For all logical processors executing in processes (OS) or guests (VMM) that contain the enclaves selected in 
step 1: 


— Issue an IPI (inter-processor interrupt) to those threads. This causes those logical processors to asynchro- 
nously exit any enclaves they might be in, and as a result flush cached linear-address to physical-address 
translations that might hold stale translations to blocked pages. There is no need for additional measures 
such as performing a “TLB shootdown”. 


4. After enclaves exit, allow logical processors to resume normal operation, including enclave re-entry as the 
tracking logic keeps track of the activity. 


5. For each page to be evicted: 


— Evict the page using the EWB leaf function with parameters include the effective-address pointer to the EPC 
page, the VA slot, a 4K byte buffer to hold the encrypted page contents, and a 128 byte buffer to hold page 
metadata. The last three elements are tied together cryptographically and must be used to later reload the 
page. 

At this point, system software has the only copy of each page data encrypted with its page metadata in main 
memory. 


34.5.4 Loading an Enclave Page 


To reload a previously evicted page, system software needs four elements: the VA slot used when the page was 
evicted, a buffer containing the encrypted page contents, a buffer containing the page metadata, and the parent 
SECS to associate this page with. If the VA page or the parent SECS are not already in the EPC, they must be 
reloaded first. 


1. Execute ELDB/ELDU (depending on the desired BLOCKED state for the page), passing as parameters: the EPC 
page linear address, the VA slot, the encrypted page, and the page metadata. 
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2. Create a mapping in the enclave context’s mapping tables (page tables and EPT tables) to allow the application 
to access that page (OS: system page table; VMM: EPT). 


The ELDB/ELDU instruction marks the VA slot empty so that the page cannot be replayed at a later date. 


34.5.5 Eviction of an SECS Page 


The eviction of an SECS page is similar to the eviction of an enclave page. The only difference is that an SECS page 
cannot be evicted until all other pages belonging to the enclave have been evicted. Since all other pages have been 
evicted, there will be no threads executing inside the enclave and tracking with ETRACK isn’t necessary. When 
reloading an enclave, the SECS page must be reloaded before all other constituent pages. 


1. Ensure all pages are evicted from enclave. 
2. Select an empty slot in a Version Array page. 
— If no VA page exists with an empty slot, create a new one using the EPA function leaf. 


3. Evict the page using the EWB leaf function with parameters include the effective-address pointer to the EPC 
page, the VA slot, a 4K byte buffer to hold the encrypted page contents and a 128 byte buffer to hold page 
metadata. The last three elements are tied together cryptographically and must be used to later reload the 


page. 


34.5.6 Eviction of a Version Array Page 


VA pages do not belong to any enclave and tracking with ETRACK isn’t necessary. When evicting the VA page, a slot 
in a different VA page must be specified in order to provide versioning of the evicted VA page. 


1. Select a slot in a Version Array page other than the page being evicted. 
— If no VA page exists with an empty slot, create a new one using the EPA leaf function. 


2. Evict the page using the EWB leaf function with parameters include the effective-address pointer to the EPC 
page, the VA slot, a 4K byte buffer to hold the encrypted page contents, and a 128 byte buffer to hold page 
metadata. The last three elements are tied together cryptographically and must be used to later reload the 
page. 


34.5.7. Allocating a Regular Page 


On processors that support SGX2, allocating a new page to an already initialized enclave is accomplished by 
invoking the EAUG leaf function. Typically, the enclave requests that the OS allocates a new page at a particular 
location within the enclave’s address space. Once allocated, the page remains in a pending state until the enclave 
executes the corresponding EACCEPT leaf function to accept the new page into the enclave. Page allocation opera- 
tions may be batched to improve efficiency. 


The typical process for allocating a regular page is as follows: 
1. Enclave requests additional memory from OS when the current allocation becomes insufficient. 
2. The OS invokes the EAUG leaf function to add a new memory page to the enclave. 
a. EAUG may only be called on a free EPC page. 
b. Successful completion of the EAUG instruction places the target page in the VALID and PENDING state. 
c. All dynamically created pages have the type PT_REG and content of all zeros. 
The OS maps the page in the enclave context's mapping tables. 


The enclave issues an EACCEPT instruction, which verifies the page’s attributes and clears the PENDING state. 
At that point the page becomes accessible for normal enclave use. 
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34.5.8 Allocating a TCS Page 


On processors that support SGX2, allocating a new TCS page to an already initialized enclave is a two-step process. 
First the OS allocates a regular page with a call to EAUG. This page must then be accepted and initialized by the 
enclave to which it belongs. Once the page has been initialized with appropriate values for a TCS page, the enclave 
requests the OS to change the page’s type to PT_TCS. This change must also be accepted. As with allocating a 
regular page, TCS allocation operations may be batched. 


A typical process for allocating a TCS page is as follows: 

1. Enclave requests an additional page from the OS. 

2. The OS invokes EAUG to add a new regular memory page to the enclave. 

a. EAUG may only be called on a free EPC page. 

b. Successful completion of the EAUG instruction places the target page in the VALID and PENDING state. 
The OS maps the page in the enclave context's mapping tables. 

The enclave issues an EACCEPT instruction, at which point the page becomes accessible for normal enclave use. 
The enclave initializes the contents of the new page. 

The enclave requests that the OS convert the page from type PT_REG to PT_TCS. 


poy Ot ees 


OS issues an EMODT instruction on the page. 
a. The parameters to EMODT indicate that the regular page should be converted into a TCS. 


b. EMODT forces all access rights to a page to be removed because TCS pages may not be accessed by enclave 
code. 


8. The enclave issues an EACCEPT instruction to confirm the requested modification. 


34.5.9 Trimming a Page 


On processors that support SGX2, Intel SGX supports the trimming of an enclave page as a special case of EMODT. 
Trimming allows an enclave to actively participate in the process of removing a page from the enclave (dealloca- 
tion) by splitting the process into first removing it from the enclave's access and then removing it from the EPC 
using the EREMOVE leaf function. The page type PT_TRIM indicates that a page has been trimmed from the 
enclave’s address space and that the page is no longer accessible to enclave software. Modifications to a page in 
the PT_TRIM state are not permitted; the page must be removed and then reallocated by the OS before the enclave 
may use the page again. Page deallocation operations may be batched to improve efficiency. 


The typical process for trimming a page from an enclave is as follows: 
1. Enclave signals OS that a particular page is no longer in use. 
2. OS invokes the EMODT leaf function on the page, requesting that the page’s type be changed to PT_TRIM. 


a. SECS and VA pages cannot be trimmed in this way, so the initial type of the page must be PT_REG or 
PT_TCS. 


b. EMODT may only be called on valid enclave pages. 


3. OS invokes the ETRACK leaf function on the enclave containing the page to track removal the TLB addresses 
from all the processors. 


4. Issue an IPI (inter-processor interrupt) to flush the stale linear-address to physical-address translations for all 
logical processors executing in processes that contain the enclave. 


Enclave issues an EACCEPT leaf function. 
The OS may now permanently remove the page from the EPC (by issuing EREMOVE). 


34.5.10 Restricting the EPCM Permissions of a Page 


On processors that support SGX2, restricting the EPCM permissions associated with an enclave page is accom- 
plished using the EMODPR leaf function. This operation requires the cooperation of the OS to flush stale entries to 
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the page and to update the page-table permissions of the page to match. Permissions restriction operations may 
be batched. 


The typical process for restricting the permissions of an enclave page is as follows: 
1. Enclave requests that the OS to restrict the permissions of an EPC page. 


2. OS performs permission restriction, flushing cached linear-address to physical-address translations, and page- 
table modifications. 


a. Invokes the EMODPR leaf function to restrict permissions (EMODPR may only be called on VALID pages). 


b. Invokes the ETRACK leaf function on the enclave containing the page to track removal of the TLB addresses 
from all the processor. 


c. Issue an IPI (inter-processor interrupt) to flush the stale linear-address to physical-address translations for 
all logical processors executing in processes that contain the enclave. 


d. Sends IPIs to trigger enclave thread exit and TLB shootdown. 

e. OS informs the Enclave that all logical processors should now see the new restricted permissions. 
3. Enclave invokes the EACCEPT leaf function. 

a. Enclave may access the page throughout the entire process. 


b. Successful call to EACCEPT guarantees that no stale cached linear-address to physical-address translations 
are present. 


34.5.11 Extending the EPCM Permissions of a Page 


On processors that support SGX2, extending the EPCM permissions associated with an enclave page is accom- 
plished directly by the enclave using the EMODPE leaf function. After performing the EPCM permission extension, 
the enclave requests the OS to update the page table permissions to match the extended permission. Security 
wise, permission extension does not require enclave threads to leave the enclave as TLBs with stale references to 
the more restrictive permissions will be flushed on demand, but to allow forward progress, an OS needs to be 
aware that an application might signal a page fault. 


The typical process for extending the permissions of an enclave page is as follows: 


1. Enclave invokes EMODPE to extend the EPCM permissions associated with an EPC page (EMODPE may only be 
called on VALID pages). 


Enclave requests that OS update the page tables to match the new EPCM permissions. 
Enclave code resumes. 


a. If cached linear-address to physical-address translations are present to the more restrictive permissions, 
the enclave thread will page fault. The SGX2-aware OS will see that the page tables permit the access and 
resume the thread, which can now successfully access the page because exiting cleared the TLB. 


b. If cached linear-address to physical-address translations are not present, access to the page with the new 
permissions will succeed without an enclave exit. 


34.5.12 VMM Oversubscription of EPC 


On processors supporting oversubscription enhancements (i.e. CPUID.(EAX=12H, ECX=0):EAX[5]=1 & EAX[6] = 
1) a Virtual Machine Monitor or other executive can more efficiently manage the EPC space available on the plat- 
form between virtualized entities. A typical process for using these instructions to support oversubscribing the 
physical EPC space on the platform is as follows: 


1. VMM creates data structures for SECS tracking including a count of child pages. 
2. VMM selects possible EPC victim pages. 


3. VMM ages the victim pages. Some of the selected pages will be accessed by the guest. In this case the VMM will 
remove these pages from the victim pool and return them to the guest. 


4. VMM makes remaining pages not present in EPT. It then issues IPI on each page to remove TLB mappings. 
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5. For every EPC victim page the VMM obtains the victim's SECS page info using ERDINFO. 


a. ENCLAVECONTEXT field in RDINFO structure will indicate the location of SECS, and the PAGE_TYPE field will 
indicate the page type. 


b. Child pages of SECS can be evicted. 

c. SECS pages may be evicted if the child count is zero. 

d. Some pages may be returned to active state depending on such things as page type or child count. 

VMM increments its evicted page count for the SECS of each page (stored in the data structure created in 1). 


7. If this is the first evicted page of that SECS, set Marker on SECS of the victim page (EINCVIRTCHILD). This 
locks the SECS in the guest. The guest cannot page out the SECS. 


EBLOCK, ETRACK, EWB eviction sequence is executed for page. 


After loading an SECS page back in, the VMM will set the correct ENCLAVECONTEXT for the guest using 
ESETCONTEXT instruction. 


34.6 CHANGES TO INSTRUCTION BEHAVIOR INSIDE AN ENCLAVE 


This section covers instructions whose behavior changes when executed in enclave mode. 


34.6.1 Illegal Instructions 


The instructions listed in Table 34-1 are ring 3 instructions which become illegal when executed inside an enclave. 
Executing these instructions inside an enclave will generate an exception. 


The first row of Table 34-1 enumerates instructions that may cause a VM exit for VMM emulation. Since a VMM 
cannot emulate enclave execution, execution of any of these instructions inside an enclave results in an invalid- 
opcode exception (#UD) and no VM exit. 


The second row of Table 34-1 enumerates I/O instructions that may cause a fault or a VM exit for emulation. Again, 
enclave execution cannot be emulated, so execution of any of these instructions inside an enclave results in #UD. 


The third row of Table 34-1 enumerates instructions that load descriptors from the GDT or the LDT or that change 
privilege level. The former class is disallowed because enclave software should not depend on the contents of the 
descriptor tables and the latter because enclave execution must be entirely with CPL = 3. Again, execution of any 
of these instructions inside an enclave results in #UD. 


The fourth row of Table 34-1 enumerates instructions that provide access to kernel information from user mode and 
can be used to aid kernel exploits from within enclave. Execution of any of these instructions inside an enclave 
results in #UD. 


Table 34-1. Illegal Instructions Inside an Enclave 


Instructions Result Comment 
CPUID, GETSEC, RDPMC, SGDT, SIDT, SLDT, STR, VMCALL, VMFUNC | #UD Might cause VM exit. 
IN, INS/INSB/INSW/INSD, OUT, OUTS/OUTSB/OUTSW/OUTSD #UD I/O fault may not safely recover. May require emulation. 


Far call, Far jump, Far Ret, INT n/INTO, IRET, LDS/LES/LFS/LGS/LSS, | #UD Access segment register could change privilege level. 
MOV to DS/ES/SS/FS/GS, POP DS/ES/SS/FS/GS, SYSCALL, 


SYSENTER 
SMSW #UD Might provide access to kernel information. 
ENCLU[EENTER], ENCLU[ERESUME] #GP Cannot enter an enclave from within an enclave. 


RDTSC and RDTSCP are legal inside an enclave for processors that support SGX2 (subject to the value of CR4.TSD). 
For processors which support SGX1 but not SGX2, RDTSC and RDTSCP will cause #UD. 


RDTSC and RDTSCP instructions may cause a VM exit when inside an enclave. 
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Software developers must take into account that the RDTSC/RDTSCP results are not immune to influences by other 
software, e.g. the TSC can be manipulated by software outside the enclave. 


34.6.2 RDRAND and RDSEED Instructions 


These instructions may cause a VM exit if the “RDRAND exiting” VM-execution control is 1. Unlike other instructions 
that can cause VM exits, these instructions are legal inside an enclave. As noted in Section 26.1 of the Inte/® 64 
and IA-32 Architectures Software Developer’s Manual, Volume 3C, any VM exit originating on an instruction 
boundary inside an enclave sets bit 27 of the exit-reason field of the VMCS. If a VMM receives a VM exit due to an 
attempt to execute either of these instructions determines (by that bit) that the execution was inside an enclave, 
it can do either of two things. It can clear the “RDRAND exiting” VM-execution control and execute VMRESUME; this 
will result in the enclave executing RDRAND or RDSEED again, and this time a VM exit will not occur. Alternatively, 
the VMM might choose to discontinue execution of this virtual machine. 


NOTE 
It is expected that VMMs that virtualize Intel SGX will not set “RDRAND exiting” to 1. 


34.6.3. PAUSE Instruction 


The PAUSE instruction may cause a VM exit from an enclave if the “PAUSE exiting” VM-execution control is 1. Unlike 
other instructions that can cause VM exits, the PAUSE instruction is legal inside an enclave. If a VMM receives a VM 
exit due to the 1-setting of “PAUSE exiting”, it can do either of two things. It can clear the “PAUSE exiting” VM- 
execution control and execute VMRESUME; this will result in the enclave executing PAUSE again, but this time aVM 
exit will not occur. Alternatively, the VMM might choose to discontinue execution of this virtual machine. 


The PAUSE instruction may also cause a VM exit outside of an enclave if the “PAUSE-loop exiting” VM-execution 
control is 1, but as the “PAUSE-loop exiting” control is ignored at CPL > 0 (see Section 24.1.3), VM exit from an 
enclave due to the 1-setting of ‘*PAUSE-LOOP exiting” will never occur. 


NOTE 
It is expected that VMMs that virtualize Intel SGX will not set “PAUSE exiting” to 1. 


34.6.4 Executions of INT1 and INT3 Inside an Enclave 


The INT1 and INT3 instructions are legal inside an enclave, however, their behavior inside an enclave differs from 
that outside an enclave. See Section 38.4.1 for details. 


34.6.5 INVD Handling when Enclaves Are Enabled 


Once processor reserved memory protections are activated (see Section 34.5), any execution of INVD will result in 
a #GP(0). 
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CHAPTER 35 
ENCLAVE EXITING EVENTS 


Certain events, such as exceptions and interrupts, incident to (but asynchronous with) enclave execution may 
cause control to transition outside of enclave mode. (Most of these also cause a change of privilege level.) To 
protect the integrity and security of the enclave, the processor will exit the enclave (and enclave mode) before 
invoking the handler for such an event. For that reason, such events are called enclave-exiting events (EEE); 
EEEs include external interrupts, non-maskable interrupts, system-management interrupts, exceptions, and VM 
exits. 


The process of leaving an enclave in response to an EEE is called an asynchronous enclave exit (AEX). To protect 
the secrecy of the enclave, an AEX saves the state of certain registers within enclave memory and then loads those 
registers with fixed values called synthetic state. 


35.1 COMPATIBLE SWITCH TO THE EXITING STACK OF AEX 


AEXs load registers with a pre-determined synthetic state. These registers may be later pushed onto the appro- 
priate stack in a form as defined by the enclave-exiting event. To allow enclave execution to resume after the 
invoking handler has processed the enclave exiting event, the asynchronous enclave exit loads the address of 
trampoline code outside of the enclave into RIP. This trampoline code eventually returns to the enclave by means 
of an ENCLU(ERESUME) leaf function. Prior to exiting the enclave the RSP and RBP registers are restored to their 
values prior to enclave entry. 


The stack to be used is chosen using the same rules as for non-SGX mode: 

¢ If there is a privilege level change, the stack will be the one associated with the new ring. 
¢ If there is no privilege level change, the current application stack is used. 

e If the IA-32e IST mechanism is used, the exit stack is chosen using that method. 


Next SSA Frame TCS 
=< ESSA 
Current SSA Frame AEP 

oe Per-Thread 
: Trampoline in URTS 
Exit Stack P 
uRSP_ — ——  encwueresume} 
Ss 
ENCLU[ERESUME] | RAX BOF 
TCS LA RBX RFLAGS 
AEP RCX CS 
RIP 


>> Error Code (optional) «—— RSP after pushes 


RSP 


Figure 35-1. Exit Stack Just After Interrupt with Stack Switch 
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In all cases, the choice of exit stack and the information pushed onto it is consistent with non-SGX operation. 
Figure 35-1 shows the Application and Exiting Stacks after an exit with a stack switch. An exit without a stack 
switch uses the Application Stack. The ERESUME leaf index value is placed into RAX, the TCS pointer is placed in 
RBX and the AEP (see below) is placed into RCX to facilitate resuming the enclave after the exit. 


Upon an AEX, the AEP (Asynchronous Exit Pointer) is loaded into the RIP. The AEP points to a trampoline code 
sequence which includes the ERESUME instruction that is later used to reenter the enclave. 


The following bits of RFLAGS are cleared before RFLAGS is pushed onto the exit stack: CF, PF, AF, ZF, SF, OF, RF. The 
remaining bits are left unchanged. 


35.2 STATE SAVING BY AEX 


The State Save Area holds the processor state at the time of an AEX. To allow handling events within the enclave 
and re-entering it after an AEX, the SSA can be a stack of multiple SSA frames as illustrated in Figure 35-2. 


SSA Stack TCS 
~<a 
GRP_N-1 
MISC_N-1 = 
CSSA 
XSAVE_N-1 OSSA 
GPR_1 
MISC_1 Current 
XSAVE 1 SSA Fram 
SECS.SSAFRAMESIZE rl crpo | ' 
(in pages) = 
MISC_0 
= XAVE_0 ' 


Figure 35-2. The SSA Stack 


The location of the SSA frames to be used is controlled by the following variables in the TCS and the SECS: 


° Size of a frame in the State Save Area (SECS.SSAFRAMESIZE): This defines the number of 4-KByte pages ina 
single frame in the State Save Area. The SSA frame size must be large enough to hold the GPR state, the XSAVE 
state, and the MISC state. 

¢ Base address of the enclave (SECS.BASEADDR): This defines the enclave's base linear address from which the 
offset to the base of the SSA stack is calculated. 

¢ Number of State Save Area Slots (TCS.NSSA): This defines the total number of slots (frames) in the State Save 
Area stack. 


* Current State Save Area Slot (TCS.CSSA): This defines the slot to use on the next exit. 
¢ State Save Area Offset (TCS.OSSA): This defines the offset of the base address of a set of State Save Area slots 
from the enclave’s base address. 


When an AEX occurs, hardware selects the SSA frame to use by examining TCS.CSSA. Processor state is saved into 
the SSA frame (see Section 35.4) and loaded with a synthetic state (as described in Section 35.3.1) to avoid 

leaking secrets, RSP and RBP are restored to their values prior to enclave entry, and TCS.CSSA is incremented. As 
will be described later, if an exception takes the last slot, it will not be possible to reenter the enclave to handle the 
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exception from within the enclave. A subsequent ERESUME restores the processor state from the current SSA 
frame and frees the SSA frame. 


The format of the XSAVE section of SSA is identical to the format used by the XSAVE/XRSTOR instructions. On 
EENTER, CSSA must be less than NSSA, ensuring that there is at least one State Save Area slot available for exits. 
If there is no free SSA frame when executing EENTER, the entry will fail. 


35.3. SYNTHETIC STATE ON ASYNCHRONOUS ENCLAVE EXIT 


35.3.1. Processor Synthetic State on Asynchronous Enclave Exit 


Table 35-1 shows the synthetic state loaded on AEX. The values shown are the lower 32 bits when the processor is 
in 32 bit mode and 64 bits when the processor is in 64 bit mode. 


Table 35-1. GPR, x87 Synthetic States on Asynchronous Enclave Exit 


Register Value 
RAX 3 (ENCLU[3] is ERESUME). 
RBX Pointer to TCS of interrupted enclave thread. 
RCX AEP of interrupted enclave thread. 
RDX, RSI, RDI 0. 
RSP Restored from SSA.uRSP. 
RBP Restored from SSA.uRBP. 
R8-R15 0 in 64-bit mode; unchanged in 32-bit mode. 
RIP AEP of interrupted enclave thread. 
RFLAGS CF, PF, AF, ZF, SF, OF, RF bits are cleared. All other bits are left unchanged. 
[x87/SSE State —~—~—*| Unless otherwise listed here, all x87 and SSE state are set to the INIT state. The INIT state is the state | 


that would be loaded by the XRSTOR instruction with bits 1:0 both set in the requested feature bitmask 
(RFBM), and both clear in XSTATE_BV the XSAVE header. 


FCW On #MF exception: set to 037EH. On all other exits: set to O37FH. 

FSW On #MF exception: set to 8081H. On all other exits: set to OH. 

MXCSR On #XM exception: set to 1F01H. On all other exits: set to 1FBOH. 

CR2 If the event that caused the AEX is a #PF, and the #PF does not directly cause a VM exit, then the low 


12 bits are cleared. 

If the #PF leads directly to a VM exit, CR2 is not updated (usual IA behavior). 

Note: The low 12 bits are not cleared if a #PF is encountered during the delivery of the EEE that caused 
the AEX. This is because the #PF was not the EEE. 


FS, GS Restored to values as of most recent EENTER/ERESUME. 


35.3.2 Synthetic State for Extended Features 


When CR4.OSXSAVE = 1, extended features (those controlled by XCRO[63:2]) are set to their respective INIT 
states when this corresponding bit of SECS.XFRM is set. The INIT state is the state that would be loaded by the 
XRSTOR instruction had the instruction mask and the XSTATE_BV field of the XSAVE header each contained the 
value XFRM. (When the AEX occurs in 32-bit mode, those features that do not exist in 32-bit mode are unchanged.) 
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35.3.3. Synthetic State for MISC Features 


State represented by SECS.MISCSELECT might also be overridden by synthetic state after it has been saved into 
the SSA. State represented by MISCSELECT[0] is not overridden but if the exiting event is a page fault then lower 
12 bits of CR2 are cleared. 


35.4 AEX FLOW 


On Enclave Exiting Events (interrupts, exceptions, VM exits or SMIs), the processor state is securely saved inside 
the enclave, a synthetic state is loaded and the enclave is exited. The EEE then proceeds in the usual exit-defined 
fashion. The following sections describes the details of an AEX: 


1. 


The exact processor state saved into the current SSA frame depends on whether the enclave is a 32-bit or a 64- 
bit enclave. In 32-bit mode (IA32_EFER.LMA = 0 || CS.L = 0), the low 32 bits of the legacy registers (EAX, EBX, 
ECX, EDX, ESP, EBP, ESI, EDI, EIP and EFLAGS) are stored. The upper 32 bits of the legacy registers and the 
64-bit registers (R8 ...R15) are not stored. 


In 64-bit mode (IA32_EFER.LMA = 1 && CS.L = 1), all 64 bits of the general processor registers (RAX, RBX, 
RCX, RDX, RSP, RBP, RSI, RDI, R8 ... R15, RIP and RFLAGS) are stored. 


The state of those extended features specified by SECS.ATTRIBUTES.XFRM are stored into the XSAVE area of 
the current SSA frame. The layout of the x87 and XMM portions (the 1st 512 bytes) depends on the current 
values of IA32_EFER.LMA and CS.L: 


If IA32_EFER.LMA = 0 || CS.L = 0, the same format (32-bit) that XSAVE/FXSAVE uses with these values. 


If IA32_EFER.LMA = 1 && CS.L = 1, the same format (64-bit) that XSAVE/FXSAVE uses with these values 
when REX.W = 1. 


ae cause of the AEX is saved in the EXITINFO field. See Table 33-10 for details and values of the various 
ields. 

The state of those miscellaneous features (see Section 33.7.2) specified by SECS.MISCSELECT are stored into 
the MISC area of the current SSA frame. 


If CET was enabled in the enclave, then the CET state of the enclave is saved in the CET state save area. If 
shadow stacks were enabled in the enclave, then the SSP is also saved into the TCS.PREVSSP field. 

Synthetic state is created for a number of processor registers to present an opaque view of the enclave state. 
Table 35-1 shows the values for GPRs, x87, SSE, FS, GS, Debug and performance monitoring on AEX. The 
synthetic state for other extended features (those controlled by XCRO[62:2]) is set to their respective INIT 
states when their corresponding bit of SECS.ATTRIBUTES. XFRM is set. The INIT state is that state as defined by 
the behavior of the XRSTOR instruction when HEADER.XSTATE_BV[n] is 0. Synthetic state of those miscella- 
neous features specified by SECS.MISCSELECT depends on the miscellaneous feature. There is no synthetic 
state required for the miscellaneous state controlled by SECS.MISCSELECT[0]. 


Any code and data breakpoints that were suppressed at the time of enclave entry are unsuppressed when 
exiting the enclave. 


RFLAGS.TF is set to the value that it had at the time of the most recent enclave entry (except for the situation 
that the entry was opt-in for debug; see Section 38.2). In the SSA, RFLAGS.TF is set to 0. 


RFLAGS.RF is set to 0 in the synthetic state. In the SSA, the value saved is the same as what would have been 
saved on stack in the non-SGX case (architectural value of RF). Thus, AEXs due to interrupts, traps, and code 
breakpoints save RF unmodified into SSA, while AEXs due to other faults save RF as 1 in the SSA. 


If the event causing AEX happened on intermediate iteration of a REP-prefixed instruction, then RF=1 is 
saved on SSA, irrespective of its priority. 

Any performance monitoring activity (including PEBS) or profiling activity (LBR, Tracing using Intel PT) on the 

exiting thread that was suppressed due to the enclave entry on that thread is unsuppressed. Any counting that 
had been demoted from AnyThread counting to MyThread counting (on one logical processor) is promoted back 
to AnyThread counting. 


The CET state of the enclosing application is restored to the state at the time of the most recent enclave entry, 
and if CET indirect branch tracking was enabled then the indirect branch tracker is unsuppressed and moved to 
the WAIT_FOR_ENDBRANCH state. 
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35.4.1 AEX Operational Detail 


Temp Variables in AEX Operational Flow 


Name Type Size (bits) Description 

TMP_RIP Effective Address 32/64 Address of instruction at which to resume execution on ERESUME. 
TMP_MODE64 binary 1 ((IA32_EFER.LMA = 1) && (CS.L = 1)). 

TMP_BRANCH_RECORD LBR Record 2x64 From/To address to be pushed onto LBR stack. 


The pseudo code in this section describes the internal operations that are executed when an AEX occurs in enclave 
mode. These operations occur just before the normal interrupt or exception processing occurs. 


(* Save RIP for later use *) 

TMP_RIP = Linear Address of Resume RIP 

(* Is the processor in 64-bit mode? *) 

TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1)); 


(* Save all registers, When saving EFLAGS, the TF bit is set to 0 and 
the RF bit is set to what would have been saved on stack in the non-SGX case *) 


IF (TMP_MODE64 = 0) 
THEN 
Save EAX, EBX, ECX, EDX, ESP, EBP, ESI, EDI, EFLAGS, EIP into the current SSA frame using 
CR_GPR_PA; (* see Table 36-5 for list of CREGs used to describe internal operation within Intel SGX *) 
SSA.RFLAGS.TF := 0; 
ELSE (* TMP_MODE64 = 1 *) 
Save RAX, RBX, RCX, RDX, RSP, RBP, RSI, RDI, R8-R15, RFLAGS, RIP into the current SSA frame using 
CR_GPR_PA; 
SSA.RFLAGS.TF := 0; 
FI; 
Save FS and GS BASE into SSA using CR_GPR_PA; 


(* store XSAVE state into the current SSA frame's XSAVE area using the physical addresses 

that were determined and cached at enclave entry time with CR_XSAVE_PAGE_i. *) 
For each XSAVE state i defined by (SECS.ATTRIBUTES.XFRM[i] = 1, destination address cached in 
CR_XSAVE_PAGE_i) 

SSA.XSAVE.i := XSAVE_STATE_i; 


(* Clear bytes 8 to 23 of XSAVE_HEADER, i.e. the next 16 bytes after XHEADER_BV *) 
CR_XSAVE_PAGE_0.XHEADER_BV[191:64] := 0; 
(* Clear bits in XHEADER_BV[63:0] that are not enabled in ATTRIBUTES. XFRM *) 
CR_XSAVE_PAGE_0.XHEADER_BV[63:0] := 
CR_XSAVE_PAGE_0.XHEADER_BV[63:0] & SECS(CR_ACTIVE_SECS).ATTRIBUTES.XFRM; 
Apply synthetic state to GPRs, RFLAGS, extended features, etc. 
(* Restore the RSP and RBP from the current SSA frame's GPR area using the physical address 
that was determined and cached at enclave entry time with CR_GPR_PA. *) 


RSP := CR_GPR_PA.URSP; 
RBP := CR_GPR_PA.URBP; 


Vol. 3D 35-5 


ENCLAVE EXITING EVENTS 


(* Restore the FS and GS *) 

FS.selector := CR_SAVE_FS.selector; 

FS.base := CR_SAVE_FS.base; 

FS.limit := CR_SAVE_FS. limit; 

FS.access_rights := CR_SAVE_FS.access_rights; 
GS.selector := CR_SAVE_GS.selector; 

GS.base := CR_SAVE_GS.base; 

GS.limit := CR_SAVE_GS.limit; 

GS.access_rights := CR_SAVE_GS.access_rights; 


(* Examine exception code and update enclave internal states*) 
exception_code := Exception or interrupt vector; 


(* Indicate the exit reason in SSA *) 
IF (exception_code = (#DE OR #DB OR #BP OR #BR OR #UD OR #MF OR #AC OR #XM )) 
THEN 
CR_GPR_PA.EXITINFO.VECTOR := exception_code; 
IF (exception code = #BP) 
THEN CR_GPR_PA.EXITINFO.EXIT_TYPE := 6; 
ELSE CR_GPR_PA.EXITINFO.EXIT_TYPE := 3; 
FI; 
CR_GPR_PA.EXITINFO.VALID := 1; 
ELSE IF (exception_code is #PF or #GP ) 
THEN 
(* Check SECS.MISCSELECT using CR_ACTIVE_SECS *) 
IF (SECS.MISCSELECT[0O] is set) 
THEN 
CR_GPR_PA.EXITINFO.VECTOR := exception_code; 
CR_GPR_PA.EXITINFO.EXIT_TYPE := 3; 
IF (exception_code is #PF) 
THEN 
SSA.MISC.EXINFO. MADDR := CR2; 
SSA.MISC.EXINFO.ERRCD := PFEC; 
SSA.MISC.EXINFO.RESERVED := 0; 
ELSE 
SSA.MISC.EXINFO. MADDR := 0; 
SSA.MISC.EXINFO.ERRCD := GPEC; 
SSA.MISC.EXINFO.RESERVED := 0; 
FI; 
CR_GPR_PA.EXITINFO.VALID := 1; 
ELSE IF (exception code is #CP) 
THEN 
IF (SECS.MISCSELECT[1] is set) 
THEN 
CR_GPR_PA.EXITINFO.VECTOR := exception_code; 
CR_GPR_PA.EXITINFO.EXIT_TYPE := 3; 
CR_GPR_PA.EXITINFO.VALID := 1; 
SSA.MISC.EXINFO. MADDR := 0; 
SSA.MISC.EXINFO.ERRCD := CPEC; 
SSA.MISC.EXINFO.RESERVED := 0; 
FI; 
FI; 
ELSE 
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CR_GPR_PA.EXITINFO.VECTOR := 0; 
CR_GPR_PA.EXITINFO.EXIT_TYPE := 0 
CR_GPR_PA.REASON.VALID := 0; 

FI; 


(* Execution will resume at the AEP *) 
RIP := CR_TCS_PA.AEP; 


(* Set EAX to the ERESUME leaf index *) 
EAX := 3; 


(* Put the TCS LA into RBX for later use by ERESUME *) 
RBX := CR_TCS_LA; 


(* Put the AEP into RCX for later use by ERESUME *) 
RCX := CR_TCS_PA.AEP; 


(* Increment the SSA frame # *) 
CR_TCS_PA.CSSA := CR_TCS_PA.CSSA + 1; 


(* Restore XCRO if needed *) 
IF (CR4.0SXSAVE = 1) 
THEN XCRO := CR_SAVE_XCRO; FI; 


Un-suppress all code breakpoints that are outside ELRANGE 


IF (CPUID.(EAX=12H, ECX=1):EAX[6]= 1) 
THEN 
IF (CR4.CET == 1 AND IA32_U_CET.SH_STK_EN == 1) 
THEN 
CR_CET_SAVE_AREA_PA.SSP := SSP; 
CR_TCS_PA.PREVSSP := SSP; 
FI; 
IF (CR4.CET == 1 AND IA32_U_CET.ENDBR_EN == 1) 
THEN 
CR_CET_SAVE_AREA_PA.TRACKER := IA32_U_CET.TRACKER; 
CR_CET_SAVE_AREA_PA.SUPPRESS := IA32_U_CET.SUPPRESS; 
FI; 
FI; 
IF ((CPUID.(EAX=7H, ECX=0):EDX[CET_IBT] = 1) OR (CPUID.(EAX=7H, ECX=0):ECX[CET_SS] = 1) 
THEN 
(* restore enclosing applications CET state *) 
IA32_U_CET := CR_SAVE_IA32_U_CET; 


IF (CPUID.(EAX=7, ECX=0):ECX[CET_SS]) 
SSP := CR_SAVE_SSP; FI; 


(* If indirect branch tracking enabled for enclosing application *) 
(* then move the tracker to wait_for_endbranch *) 
IF (CR4.CET == 1 AND IA32_U_CET.ENDBR_EN == 1) 
THEN 
IA32_U_CET.TRACKER := WAIT_FOR_ENDBRANCH; 
IA32_U_CET.SUPPRESS := 0; 
FI; 
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FI; 


(* Update the thread context to show not in enclave mode *) 
CR_ENCLAVE_MODE := 0; 


(* Assure consistent translations. *) 
Flush linear context including TLBs and paging-structure caches 


IF (CR_DBGOPTIN = 0) 
THEN 
Un-suppress all breakpoints that overlap ELRANGE 
(* Clear suppressed breakpoint matches *) 
Restore suppressed breakpoint matches 
(* Restore TF *) 
RFLAGS.TF := CR_SAVE_TF; 
Un-suppress monitor trap flag; 
Un-suppress branch recording facilities; 
Un-suppress all suppressed performance monitoring activity; 
Promote any sibling-thread counters that were demoted from AnyThread to MyThread during enclave 
entry back to AnyThread; 
FI; 


IF the “monitor trap flag” VM-execution control is 1 
THEN Pend MTF VM Exit at the end of exit; FI; 


(* Clear low 12 bits of CR2 on #PF *) 
IF (Exception code is #PF) 
THEN CR2 := CR2 & ~OxFFF; FI; 


(* end_of_flow *) 


(* Execution continues with normal event processing. *) 
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CHAPTER 36 


INTEL® SGX INSTRUCTION REFERENCES 


This chapter describes the supervisor and user level instructions provided by Intel® Software Guard Extensions 
(Intel® SGX). In general, various functionality is encoded as leaf functions within the ENCLS (supervisor), ENCLU 
(user), and the ENCLV (virtualization operation) instruction mnemonics. Different leaf functions are encoded by 
specifying an input value in the EAX register of the respective instruction mnemonic. 


36.1 


INTEL® SGX INSTRUCTION SYNTAX AND OPERATION 


ENCLS, ENCLU and ENCLV instruction mnemonics for all leaf functions are covered in this section. 


For all instructions, the value of CS.D is ignored; addresses and operands are 64 bits in 64-bit mode and are other- 
wise 32 bits. Aside from EAX specifying the leaf number as input, each instruction leaf may require all or some 

subset of the RBX/RCX/RDxX as input parameters. Some leaf functions may return data or status information in one 
or more of the general purpose registers. 


36.1.1 


ENCLS Register Usage Summary 


Table 36-1 summarizes the implicit register usage of supervisor mode enclave instructions. 


Table 36-1. Register Usage of Privileged Enclave Instruction Leaf Functions 


Instr. Leaf EAX RBX RCX RDX 

ECREATE OOH (In) PAGEINFO (In, EA) EPCPAGE (In, EA) 

EADD 01H (In) PAGEINFO (In, EA) EPCPAGE (In, EA) 

EINIT O2H (In) SIGSTRUCT (In, EA) SECS (In, EA) EINITTOKEN (In, EA) 
EREMOVE O3H (In) EPCPAGE (In, EA) 

EDBGRD O4H (In) Result Data (Out) EPCPAGE (In, EA) 

EDBGWR OSH (In) Source Data (In) EPCPAGE (In, EA) 

EEXTEND O6H (In) SECS (In, EA) EPCPAGE (In, EA) 

ELDB O7H (In) PAGEINFO (In, EA) EPCPAGE (In, EA) VERSION (In, EA) 
ELDU OSH (In) PAGEINFO (In, EA) EPCPAGE (In, EA) VERSION (In, EA) 
EBLOCK OSH (In) EPCPAGE (In, EA) 

EPA OAH (In) PT_VA (In) EPCPAGE (In, EA) 

EWB OBH (In) PAGEINFO (In, EA) EPCPAGE (In, EA) VERSION (In, EA) 
ETRACK OCH (In) EPCPAGE (In, EA) 

EAUG ODH (In) PAGEINFO (In, EA) EPCPAGE (In, EA) 

EMODPR OEH (In) SECINFO (In, EA) EPCPAGE (In, EA) 

EMODT OFH (In) SECINFO (In, EA) EPCPAGE (In, EA) 

ERDINFO 010H (In) RDINFO (In, EA*) EPCPAGE (In, EA) 

ETRACKC 011H (In) EPCPAGE (In, EA) 

ELDBC 012H (In) PAGEINFO (In, EA*) EPCPAGE (In, EA) VERSION (In, EA) 
ELDUC 013H (In) PAGEINFO (In, EA*) EPCPAGE (In, EA) VERSION (In, EA) 
EA: Effective Address 
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36.1.2 ENCLU Register Usage Summary 


Table 36-2 summarizes the implicit register usage of user mode enclave instructions. 


Table 36-2. Register Usage of Unprivileged Enclave Instruction Leaf Functions 


Instr. Leaf EAX RBX RCX RDX 
EREPORT OOH (In) TARGETINFO (In, EA) REPORTDATA (In, EA) OUTPUTDATA (In, EA) 
EGETKEY 01H (In) KEYREQUEST (In, EA) KEY (In, EA) 
EENTER O2H (In) TCS (In, EA) AEP (In, EA) 
RBX.CSSA (Out) Return (Out, EA) 
ERESUME O3H (In) TCS (In, EA) AEP (In, EA) 
EEXIT O4H (In) Target (In, EA) Current AEP (Out) 
EACCEPT OSH (In) SECINFO (In, EA) EPCPAGE (In, EA) 
EMODPE OGH (In) SECINFO (In, EA) EPCPAGE (In, EA) 
EACCEPTCOPY | 07H (In) SECINFO (In, EA) EPCPAGE (In, EA) EPCPAGE (In, EA) 
EA: Effective Address 


36.1.3. ENCLV Register Usage Summary 


Table 36-3 summarizes the implicit register usage of virtualization operation enclave instructions. 


Table 36-3. Register Usage of Virtualization Operation Enclave Instruction Leaf Functions 


Instr. Leaf EAX RBX RCX RDX 

EDECVIRTCHILD | OOH (In) EPCPAGE (In, EA) SECS (In, EA) 

EINCVIRTCHILD | 01H (In) EPCPAGE (In, EA) SECS (In, EA) 

ESETCONTEXT | 02H (In) EPCPAGE (In, EA) Context Value (In, EA) 
EA: Effective Address 


36.1.4 Information and Error Codes 


Information and error codes are reported by various instruction leaf functions to show an abnormal termination of 
the instruction or provide information which may be useful to the developer. Table 36-4 shows the various codes 
and the instruction which generated the code. Details of the meaning of the code is provided in the individual 
instruction. 


Table 36-4. Error or Information Codes for Intel® SGX Instructions 


Name Value Returned By 

No Error 0 

SGX_INVALID_SIG_STRUCT 1 EINIT 

SGX_INVALID_ATTRIBUTE 2 EINIT, EGETKEY 

SGX_BLKSTATE 3 EBLOCK 

SGX_INVALID_MEASUREMENT 4 EINIT 

SGX_NOTBLOCKABLE 5 EBLOCK 

SGX_PG_INVLD 6 EBLOCK, ERDINFO, ETRACKC 

SGX_EPC_PAGE_CONFLICT 7 EBLOCK, EMODPR, EMODT, ERDINFO , EDECVIRTCHILD, EINCVIRTCHILD, ELDBC, 
ELDUC, ESETCONTEXT, ETRACKC 
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Table 36-4. Error or Information Codes for Intel® SGX Instructions 


Name Value Returned By 
SGX_INVALID_SIGNATURE 8 EINIT 
SGX_MAC_COMPARE_FAIL 9 ELDB, ELDU, ELDBC, ELDUC 
SGX_PAGE_NOT_BLOCKED 10 EWB 
SGX_NOT_TRACKED 11 EWB, EACCEPT 
SGX_VA_SLOT_OCCUPIED 12 EWB 
SGX_CHILD_PRESENT 13 EWB, EREMOVE 
SGX_ENCLAVE_ACT 14 EREMOVE 
SGX_ENTRYEPOCH_LOCKED 15 EBLOCK 
SGX_INVALID_EINITTOKEN 16 EINIT 
SGX_PREV_TRK_INCMPL 17 ETRACK, ETRACKC 
SGX_PG_IS_SECS 18 EBLOCK 
SGX_PAGE_ATTRIBUTES_MISMATCH | 19 EACCEPT, EACCEPTCOPY 
SGX_PAGE_NOT_MODIFIABLE 20 EMODPR, EMODT 
SGX_PAGE_NOT_DEBUGGABLE 21 EDBGRD, EDBGWR 
SGX_INVALID_COUNTER 25 EDECVIRTCHILD 
SGX_PG_NONEPC 26 ERDINFO 
SGX_TRACK_NOT_REQUIRED 27 ETRACKC 
SGX_INVALID_CPUSVN 32 EINIT, EGETKEY 
SGX_INVALID_ISVSVN 64 EGETKEY 
SGX_UNMASKED_EVENT 128 EINIT 
SGX_INVALID_KEYNAME 256 EGETKEY 


36.1.5 Internal CREGs 


The CREGs as shown in Table 5-4 are hardware specific registers used in this document to indicate values kept by 
the processor. These values are used while executing in enclave mode or while executing an Intel SGX instruction. 
These registers are not software visible and are implementation specific. The values in Table 36-5 appear at various 
places in the pseudo-code of this document. They are used to enhance understanding of the operations. 


Table 36-5. List of Internal CREG 


Name Size (Bits) Scope 
CR_ENCLAVE_MODE 1 LP 
CR_DBGOPTIN 1 LP 
CR_TCS_LA 64 LP 
CR_TCS_PA 64 LP 
CR_ACTIVE_SECS 64 LP 
CR_ELRANGE 128 LP 
CR_SAVE_TF 1 LP 
CR_SAVE_FS 64 LP 
CR_GPR_PA 64 LP 
CR_XSAVE_PAGE_n 64 LP 
CR_SAVE_DR7 64 LP 
CR_SAVE_PERF_GLOBAL_CTRL 64 LP 
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Table 36-5. List of Internal CREG 


Name Size (Bits) Scope 
CR_SAVE_DEBUGCTL 64 LP 
CR_SAVE_PEBS_ENABLE 64 LP 
CR_CPUSVN 128 PACKAGE 
CR_SGXOWNEREPOCH 128 PACKAGE 
CR_SAVE_XCRO 64 LP 
CR_SGX_ATTRIBUTES_MASK 128 LP 
CR_PAGING_VERSION 64 PACKAGE 
CR_VERSION_THRESHOLD 64 PACKAGE 
CR_NEXT_EID 64 PACKAGE 
CR_BASE_PK 128 PACKAGE 
CR_SEAL_FUSES 128 PACKAGE 
CR_CET_SAVE_AREA_PA 64 LP 
CR_ENCLAVE_SS_TOKEN_PA 64 LP 
CR_SAVE_IA32_U_CET 64 LP 
CR_SAVE_SSP 64 LP 


36.1.6 Concurrent Operation Restrictions 


Under certain conditions, Intel SGX disallows certain leaf functions from operating concurrently. Listed below are 
some examples of concurrency that are not allowed. 


¢ For example, Intel SGX disallows the following leafs to concurrently operate on the same EPC page. 


— ECREATE, EADD, and EREMOVE are not allowed to operate on the same EPC page concurrently with 
themselves. 


— EADD, EEXTEND, and EINIT leaves are not allowed to operate on the same SECS concurrently. 
¢ Intel SGX disallows the EREMOVE leaf from removing pages from an enclave that is in use. 


¢ Intel SGX disallows entry (EENTER and ERESUME) to an enclave while a page from that enclave is being 
removed. 


When disallowed operation is detected, a leaf function may do one of the following: 
¢ Return an SGX_EPC_PAGE_CONFLICT error code in RAX. 
* Cause a #GP(0) exception. 


To prevent such exceptions, software must serialize leaf functions or prevent these leaf functions from accessing 
the same EPC page. 


36.1.6.1 Concurrency Tables of Intel® SGX Instructions 


The tables below detail the concurrent operation restrictions of all SGX leaf functions. For each leaf function, the 
table has a separate line for each of the EPC pages the leaf function accesses. 


For each such EPC page, the base concurrency requirements are detailed as follows: 


° Exclusive Access means that no other leaf function that requires either shared or exclusive access to the 
same EPC page may be executed concurrently. For example, EADD requires an exclusive access to the target 
page it accesses. 


¢ Shared Access means that no other leaf function that requires an exclusive access to the same EPC page may 
be executed concurrently. Other leaf functions that require shared access may run concurrently. For example, 
EADD requires a shared access to the SECS page it accesses. 
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¢ Concurrent Access means that any other leaf function that requires any access to the same EPC page may be 
executed concurrently. For example, EGETKEY has no concurrency requirements for the KEYREQUEST page. 


In addition to the base concurrency requirements, additional concurrency requirements are listed, which apply 
only to specific sets of leaf functions. For example, there are additional requirements that apply for EADD, EXTEND 
and EINIT. EADD and EEXTEND can't execute concurrently on the same SECS page. 


The tables also detail the leaf function's behavior when a conflict happens, i.e., a concurrency requirement is not 
met. In this case, the leaf function may return an SGX_EPC_PAGE_CONFLICT error code in RAX, or it may cause an 
exception. In addition, the tables detail those conflicts where a VM Exit may be triggered, and list the Exit Qualifi- 
cation code that is provided in such cases. 


Table 36-6. Base Concurrency Restrictions 


Base Concurrency Restrictions 
wile eee Access On Conflict ration 
EACCEPT Target [DS:RCX] Shared #GP 
SECINFO [DS:RBX] Concurrent 
EACCEPTCOPY Target [DS:RCX] Concurrent 
Source [DS:RDX] Concurrent 
SECINFO [DS:RBX] Concurrent 
[EADD —S*(Y| Target ~=3—S—~C«<CS(DSRCX]~—~Cté«‘C«s«C«*YCUEXClive’”«3F}) Ss VG) =——~S~S~S*=é«~CERC_PAAGE_CONFLICT_EXCEPTION | 
SECS [DS:RBX]PAGEINFO. | Shared #GP 
SECS 
EAUG Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
SECS [DS:RBX]PAGEINFO. | Shared #GP 
SECS 
EBLOCK Target [DS:RCX] Shared SGX_EPC_PAGE 
_CONFLICT 
[ECREATE —*«([SECS-~—<sé‘isé~dY(SRCXY”~O~*~‘;‘;S;*C‘*CExclusive’:«=3OF})©) sf #GP)=——S~*~=«~rCERC_PAAGE_CONFLICT_EXCCEPTIION | 
EDBGRD Target [DS:RCX] Shared #GP 
EDBGWR Target [DS:RCX] Shared #GP 
EDECVIRTCHILD | Target [DS:RBX] Shared SGX_EPC_PAGE 
_CONFLICT 
SECS [DS:RCX] Concurrent 
EENTERTCS SECS [DS:RBX] Shared #GP 
EEXIT Concurrent 
EEXTEND Target [DS:RCX] Shared #GP 
SECS [DS:RBX] Concurrent 
EGETKEY KEYREQUEST | [DS:RBX] Concurrent 
OUTPUTDATA | [DS:RCX] Concurrent 
EINCVIRTCHILD | Target [DS:RBX] Shared SGX_EPC_PAGE 
_CONFLICT 
SECS [DS:RCX] Concurrent 
EINIT SECS [DS:RCX] Shared #GP 
ELDB/ELDU Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
[VA ~—~—SdY([DS‘RDX]—“‘<‘(<rdSSHareds=~=—=—~~SCOHOPté“~;‘<~dS*é‘(‘(((NSSNNCOO NY 
SECS [DS:RBX]PAGEINFO. | Shared #GP 
SECS 
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Table 36-6. Base Concurrency Restrictions 


Base Concurrency Restrictions 
veal Parameter Access On Conflict Se nic ctlon Exit 
EDLBC/ELDUC Target [DS:RCX] Exclusive SGX_EPC_PAGE | EPC_PAGE_CONFLICT_ERROR 
_CONFLICT 
VA [DS:RDX] Shared SGX_EPC_PAGE 
_CONFLICT 
SECS [DS:RBX]PAGEINFO. | Shared SGX_EPC_PAGE 
SECS _CONFLICT 
EMODPE Target [DS:RCX] Concurrent 
SECINFO [DS:RBX] Concurrent 
EMODPR Target [DS:RCX] Shared #GP 
EMODT Target [DS:RCX] Exclusive SGX_EPC_PAGE | EPC_PAGE_CONFLICT_ERROR 
_CONFLICT 
EPA VA [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
ERDINFO Target [DS:RCX] Shared SGX_EPC_PAGE 
_CONFLICT 
EREMOVE Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
EREPORT TARGETINFO [DS:RBX] Concurrent 
REPORTDATA _ | [DS:RCX] Concurrent 
OUTPUTDATA | [DS:RDX] Concurrent 
ERESUME TCS [DS:RBX] Shared #GP 
ESETCONTEXT | SECS [DS:RCX] Shared SGX_EPC_PAGE 
_CONFLICT 
ETRACK SECS [DS:RCX] Shared #GP 
ETRACKC Target [DS:RCX] Shared SGX_EPC_PAGE 
_CONFLICT 
SECS Implicit Concurrent 
EWB Source [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
VA [DS:RDX] Shared #GP 
Table 36-7. Additional Concurrency Restrictions 
Additional Concurrency Restrictions 
vs. EACCEPT, 
Leaf Parameter eMoore ODER: “ ee NSHETRACK TRAE 
EMODT 
a catia BECESS conti RELESS a aick 
EACCEPT Target [DS:RCX] Exclusive #GP Concurrent Concurrent 
SECINFO [DS:RBX] Concurrent Concurrent Concurrent 
EACCEPTCOPY | Target [DS:RCX] Exclusive #GP Concurrent Concurrent 
Source [DS:RDX] Concurrent Concurrent Concurrent 
SECINFO [DS:RBX] Concurrent Concurrent Concurrent 
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Table 36-7. Additional Concurrency Restrictions 


Additional Concurrency Restrictions 
vs. EACCEPT, 
Leaf Parameter aobpe. eNobEn "° re Poe here a inene 
EMODT 
EEESS soatiek REEESS cunt AREESS entice 
EADD Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGEINFO. | Concurrent Exclusive #OP Concurrent 
SECS 
EAUG Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGEINFO. | Concurrent Concurrent Concurrent 
SECS 
EBLOCK Target [DS:RCX] Concurrent Concurrent Concurrent 
ECREATE SECS [DS:RCX] Concurrent Concurrent Concurrent 
EDBGRD Target [DS:RCX] Concurrent Concurrent Concurrent 
EDBGWR Target [DS:RCX] Concurrent Concurrent Concurrent 
EDECVIRTCHILD | Target [DS:RBX] Concurrent Concurrent Concurrent 
SECS [DS:RCX] Concurrent Concurrent Concurrent 
EENTERTCS SECS [DS:RBX] Concurrent Concurrent Concurrent 
EEXIT Concurrent Concurrent Concurrent 
EEXTEND Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS [DS:RBX] Concurrent Exclusive #OP Concurrent 
EGETKEY KEYREQUEST | [DS:RBX] Concurrent Concurrent Concurrent 
OUTPUTDATA | [DS:RCX] Concurrent Concurrent Concurrent 
EINCVIRTCHILD | Target [DS:RBX] Concurrent Concurrent Concurrent 
SECS [DS:RCX] Concurrent Concurrent Concurrent 
EINIT SECS [DS:RCX] Concurrent Exclusive #OP Concurrent 
ELDB/ELDU Target [DS:RCX] Concurrent Concurrent Concurrent 
VA [DS:RDX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGEINFO. | Concurrent Concurrent Concurrent 
SECS 
EDLBC/ELDUC Target [DS:RCX] Concurrent Concurrent Concurrent 
VA [DS:RDX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGEINFO. | Concurrent Concurrent Concurrent 
SECS 
EMODPE Target [DS:RCX] Exclusive #GP Concurrent Concurrent 
SECINFO [DS:RBX] Concurrent Concurrent Concurrent 
EMODPR Target [DS:RCX] Exclusive SGX_EPC_ | Concurrent Concurrent 
PAGE_CON 
FLICT 
EMODT Target [DS:RCX] Exclusive SGX_EPC_ | Concurrent Concurrent 
PAGE_CON 
FLICT 
EPA VA [DS:RCX] Concurrent Concurrent Concurrent 
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Table 36-7. Additional Concurrency Restrictions 


Additional Concurrency Restrictions 
vs. EACCEPT, 
Leaf Parameter SMOBRE) SHODER: = eae Ms ETRAGH ETRAERE 
EMODT 
AIEESS sai maces ral BECESS eatiet 
ERDINFO Target [DS:RCX] Concurrent Concurrent Concurrent 
EREMOVE Target [DS:RCX] Concurrent Concurrent Concurrent 
EREPORT TARGETINFO | [DS:RBX] Concurrent Concurrent Concurrent 
REPORTDATA | [DS:RCX] Concurrent Concurrent Concurrent 
OUTPUTDATA | [DS:RDX] Concurrent Concurrent Concurrent 
ERESUME TCS [DS:RBX] Concurrent Concurrent Concurrent 
ESETCONTEXT | SECS [DS:RCX] Concurrent Concurrent Concurrent 
ETRACK SECS [DS:RCX] Concurrent Concurrent Exclusive SGX_EPC_ 
PAGE_CO 
NFLICT! 
ETRACKC Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS Implicit Concurrent Concurrent Exclusive SGX_EPC_ 
PAGE_CO 
NFLICT! 
EWB Source [DS:RCX] Concurrent Concurrent Concurrent 
VA [DS:RDX] Concurrent Concurrent Concurrent 
NOTES: 


1. SGX_CONFLICT VM Exit Qualification =TRACKING_RESOURCE_CONFLICT. 
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ENCLS—Execute an Enclave System Function of Specified Leaf Number 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
NP OF 01 CF ZO VV NA This instruction is used to execute privileged Intel SGX leaf func- 
ENCLS tions that are used for managing and debugging the enclaves. 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Implicit Register Operands 
ZO NA NA NA See Section 36.3 
Description 


The ENCLS instruction invokes the specified privileged Intel SGX leaf function for managing and debugging 
enclaves. Software specifies the leaf function by setting the appropriate value in the register EAX as input. The 
registers RBX, RCX, and RDX have leaf-specific purpose, and may act as input, as output, or may be unused. In 64- 
bit mode, the instruction ignores upper 32 bits of the RAX register. 

The ENCLS instruction produces an invalid-opcode exception (#UD) if CRO.PE = 0 or RFLAGS.VM = 1, or if itis 
executed in system-management mode (SMM). Additionally, any attempt to execute the instruction when CPL > 0 
results in #UD. The instruction produces a general-protection exception (#GP) if CRO.PG = 0 or if an attempt is 
made to invoke an undefined leaf function. 

In VMX non-root operation, execution of ENCLS may cause a VM exit if the “enable ENCLS exiting” VM-execution 
control is 1. In this case, execution of individual leaf functions of ENCLS is governed by the ENCLS-exiting bitmap 
field in the VMCS. Each bit in that field corresponds to the index of an ENCLS leaf function (as provided in EAX). 
Software in VMX root operation can thus intercept the invocation of various ENCLS leaf functions in VMX non-root 
operation by setting the “enable ENCLS exiting” VM-execution control and setting the corresponding bits in the 
ENCLS-exiting bitmap. 

Addresses and operands are 32 bits outside 64-bit mode (IA32_EFER.LMA = 0 || CS.L = 0) and are 64 bits in 64- 
bit mode (IA32_EFER.LMA = 1 || CS.L = 1). CS.D value has no impact on address calculation. The DS segment is 
used to create linear addresses. 


Segment override prefixes and address-size override prefixes are ignored, and is the REX prefix in 64-bit mode. 


Operation 


IF TSX_ACTIVE 
THEN GOTO TSX_ABORT_PROCESSING; FI; 


IF CRO.PE = 0 or RFLAGS.VM = 1 or in SMM or CPUID.SGX_LEAF.O:EAX.SE1 = 0 
THEN #UD; Fl; 


IF (CPL > 0) 
THEN #UD; Fl; 


IF in VMX non-root operation and the “enable ENCLS exiting” VM-execution control is 1 


THEN 
IF EAX < 63 and ENCLS_exiting_bitmap[EAX] = 1 or EAX> 62 and ENCLS_exiting_bitmap[63] = 1 
THEN VM exit; 
Fl; 


Fl; 
IF IA32_FEATURE_CONTROL.LOCK = O or IA32_FEATURE_CONTROL.SGX_ENABLE = 0 
THEN #GP(0); Fl; 


IF (EAX is an invalid leaf number) 
THEN #GP(0); Fl; 
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IF CRO.PG = 0 
THEN #GP(0); Fl; 


(* DS must not be an expanded down segment *) 
IF not in 64-bit mode and DS.Type is expand-down data 
THEN #GP(0); Fl; 


Jump to leaf specific flow 


Flags Affected 
See individual leaf functions 


Protected Mode Exceptions 

#UD If any of the LOCK/66H/REP/VEX prefixes are used. 
If current privilege level is not 0. 
If CPUID.(EAX=12H,ECX=0):EAX.SGX1 [bit 0] = 0. 
If logical processor is in SMM. 

#GP(0) If IA32_FEATURE_CONTROL.LOCK = 0. 
If IA32_FEATURE_CONTROL.SGX_ENABLE = 0. 
If input value in EAX encodes an unsupported leaf. 
If data segment expand down. 
If CRO.PG=0. 


Real-Address Mode Exceptions 
#UD ENCLS is not recognized in real mode. 


Virtual-8086 Mode Exceptions 
#UD ENCLS is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
Same exceptions as in protected mode. 


64-Bit Mode Exceptions 

#UD If any of the LOCK/66H/REP/VEX prefixes are used. 
If current privilege level is not 0. 
If CPUID.(EAX=12H,ECX=0):EAX.SGX1 [bit 0] = 0. 
If logical processor is in SMM. 

#GP(0) If IA32_FEATURE_CONTROL.LOCK = 0. 
If IA32_FEATURE_CONTROL.SGX_ENABLE = 0. 
If input value in EAX encodes an unsupported leaf. 
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ENCLU—Execute an Enclave User Function of Specified Leaf Number 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
NP OF 01 D7 ZO VIV NA This instruction is used to execute non-privileged Intel SGX leaf 
ENCLU functions. 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Implicit Register Operands 
ZO NA NA NA See Section 36.4 
Description 


The ENCLU instruction invokes the specified non-privileged Intel SGX leaf functions. Software specifies the leaf 
function by setting the appropriate value in the register EAX as input. The registers RBX, RCX, and RDX have leaf- 
specific purpose, and may act as input, as output, or may be unused. In 64-bit mode, the instruction ignores upper 
32 bits of the RAX register. 

The ENCLU instruction produces an invalid-opcode exception (#UD) if CRO.PE = 0 or RFLAGS.VM = 1, or if it is 
executed in system-management mode (SMM). Additionally, any attempt to execute this instruction when CPL < 3 
results in #UD. The instruction produces a general-protection exception (#GP) if either CRO.PG or CRO.NE is 0, or 
if an attempt is made to invoke an undefined leaf function. The ENCLU instruction produces a device not available 
exception (#NM) if CRO.TS = 1. 

Addresses and operands are 32 bits outside 64-bit mode (IA32_EFER.LMA = 0 or CS.L = 0) and are 64 bits in 64- 
bit mode (IA32_EFER.LMA = 1 and CS.L = 1). CS.D value has no impact on address calculation. The DS segment 
is used to create linear addresses. 


Segment override prefixes and address-size override prefixes are ignored, as is the REX prefix in 64-bit mode. 


Operation 


IN_64BIT_MODE := 0; 
IF TSX_ACTIVE 
THEN GOTO TSX_ABORT_PROCESSING; FI; 


(* If enclosing app has CET indirect branch tracking enabled then if it is not ERESUME leaf cause a #CP fault *) 
(* If the ERESUME is not successful it will leave tracker in WAIT_FOR_ENDBRANCH *) 
TRACKER = (CPL == 3) ? IA32_U_CET. TRACKER : IA32_S_CET.TRACKER 
IF EndbranchEnabledAndNotSuppressed(CPL) and TRACKER = WAIT_FOR_ENDBRANCH and 

(EAX != ERESUME or CRO.TS or (in SMM) or (CPUID.SGX_LEAF.0:EAX.SE1 = 0) or (CPL < 3)) 

THEN 
Handle CET State machine violation (* see Section 18.3.6, “Legacy Compatibility Treatment” in the 
Intel® 64 and IA-32 Architectures Software Developer's Manual, Volume 1. *) 

Fl; 


IF CRO.PE= 0 or RFLAGS.VM = 1 or in SMM or CPUID.SGX_LEAF.0:EAX.SE1 = 0 
THEN #UD; Fl; 


IF CRO.TS = 1 
THEN #NM; FI; 


IF CPL < 3 
THEN #UD; Fl; 


IF IA32_FEATURE_CONTROL.LOCK = 0 or IA32_FEATURE_CONTROLSGX_ENABLE = 0 
THEN #GP(0); Fl; 
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IF EAX is invalid leaf number 
THEN #GP(0); Fl; 


IF CRO.PG = 0 or CRO.NE = 0 
THEN #GP(0); Fl; 


IN_64BIT_MODE := IA32_EFER.LMA AND CS.L? 1 : 0; 
(* Check not in 16-bit mode and DS is not a 16-bit segment *) 
IF not in 64-bit mode and CS.D = 0 

THEN #GP(0); Fl; 


IF CR_ENCLAVE_MODE = 1 and (EAX = 2 or EAX = 3) (* EENTER or ERESUME *) 
THEN #GP(0); Fl; 


IF CR_ENCLAVE_MODE = 0 and (EAX = 0 or EAX = 1 or EAX = 4 or EAX = 5 or EAX = 6 or EAX = 7) 
(* EREPORT, EGETKEY, EEXIT, EACCEPT, EMODPE, or EACCEPTCOPY *) 
THEN #GP(0); Fl; 


Jump to leaf specific flow 


Flags Affected 
See individual leaf functions 


Protected Mode Exceptions 
#UD If any of the LOCK/66H/REP/VEX prefixes are used. 
If current privilege level is not 3. 
If CPUID.(EAX=12H,ECX=0):EAX.SGX1 [bit 0] = 0. 
If logical processor is in SMM. 
#GP(0) If IA32_FEATURE_CONTROL.LOCK = 0. 
If IA32_FEATURE_CONTROL.SGX_ENABLE = 0. 
If input value in EAX encodes an unsupported leaf. 
If input value in EAX encodes EENTER/ERESUME and ENCLAVE_MODE = 1. 


If input value in EAX encodes EGETKEY/EREPORT/EEXIT/EACCEPT/EACCEPTCOPY/EMODPE 
and ENCLAVE_MODE = 0. 


If operating in 16-bit mode. 

If data segment is in 16-bit mode. 

If CRO.PG = 0 or CRO.NE= 0. 
#NM If CRO.TS = 1. 


Real-Address Mode Exceptions 
#UD ENCLS is not recognized in real mode. 


Virtual-8086 Mode Exceptions 
#UD ENCLS is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 
Same exceptions as in protected mode. 


36-12 Vol. 3D 


INTEL® SGX INSTRUCTION REFERENCES 


64-Bit Mode Exceptions 
#UD If any of the LOCK/66H/REP/VEX prefixes are used. 
If current privilege level is not 3. 
If CPUID.(EAX=12H,ECX=0):EAX.SGX1 [bit 0] = 0. 
If logical processor is in SMM. 
#GP(0) If IA32_FEATURE_CONTROL.LOCK = 0. 
If IA32_FEATURE_CONTROL.SGX_ENABLE = 0. 
If input value in EAX encodes an unsupported leaf. 
If input value in EAX encodes EENTER/ERESUME and ENCLAVE_MODE = 1. 


If input value in EAX encodes EGETKEY/EREPORT/EEXIT/EACCEPT/EACCEPTCOPY/EMODPE 
and ENCLAVE_MODE = 0. 


If CRO.NE= 0. 
#NM If CRO.TS = 1. 
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ENCLV—Execute an Enclave VMM Function of Specified Leaf Number 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
NP OF 01 CO ZO VIN NA This instruction is used to execute privileged SGX leaf functions 
ENCLV that are reserved for VMM use. They are used for managing the 
enclaves. 


Instruction Operand Encoding 


Op/En Operand 1 Operand 2 Operand 3 Implicit Register Operands 
ZO NA NA NA See Section 36.3 
Description 


The ENCLV instruction invokes the virtualization SGX leaf functions for managing enclaves in a virtualized environ- 
ment. Software specifies the leaf function by setting the appropriate value in the register EAX as input. The regis- 
ters RBX, RCX, and RDX have leaf-specific purpose, and may act as input, as output, or may be unused. In non 64- 
bit mode, the instruction ignores upper 32 bits of the RAX register. 

The ENCLV instruction produces an invalid-opcode exception (#UD) if CRO.PE = 0 or RFLAGS.VM = 1, if it is 
executed in system-management mode (SMM), or not in VMX operation. Additionally, any attempt to execute the 
instruction when CPL > 0 results in #UD. The instruction produces a general-protection exception (#GP) if CRO.PG 
= 0 or if an attempt is made to invoke an undefined leaf function. 

Software in VMX root mode of operation can enable execution of the ENCLV instruction in VMX non-root mode by 
setting enable ENCLV execution control in the VMCS. If enable ENCLV execution control in the VMCS is clear, execu- 
tion of the ENCLV instruction in VMX non-root mode results in #UD. 

When execution of ENCLV instruction in VMX non-root mode is enabled, software in VMX root operation can inter- 
cept the invocation of various ENCLV leaf functions in VMX non-root operation by setting the corresponding bits in 
the ENCLV-exiting bitmap. 

Addresses and operands are 32 bits in 32-bit mode (IA32_EFER.LMA == 0 || CS.L == 0) and are 64 bits in 64-bit 
mode (IA32_EFER.LMA == 1 && CS.L == 1). CS.D value has no impact on address calculation. 


Segment override prefixes and address-size override prefixes are ignored, as is the REX prefix in 64-bit mode. 


Operation 


IF TSX_ACTIVE 
THEN GOTO TSX_ABORT_PROCESSING; FI; 


IF CRO.PE = 0 or RFLAGS.VM = 1 or in SMM or CPUID.SGX_LEAF.0:EAX.OSS = 0 
THEN #UD; FI; 


IF not in VMX Operation or (IA32_EFER.LMA = 1 and CS.L = 0) 
THEN #UD; FI; 


IF (CPL > 0) 
THEN #UD; Fl; 


IF in VMX non-root operation 
IF “enable ENCLV exiting” VM-execution control is 1 
THEN 
IF EAX < 63 and ENCLV_exiting_bitmap[EAX] = 1 or EAX> 62 and ENCLV_exiting_bitmap[63] = 1 
THEN VM exit; 
Fl; 
ELSE 
#UD; Fl; 
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Fl; 


IF IA32_FEATURE_CONTROL.LOCK = 0 or IA32_FEATURE_CONTROL.SGX_ENABLE = O 


THEN #GP(0); Fl; 


IF (EAX is an invalid leaf number) 
THEN #GP(0); FI; 


IF CRO.PG = 0 
THEN #GP(0); Fl; 


(* DS must not be an expanded down segment *) 
IF not in 64-bit mode and DS.Type is expand-down data 
THEN #GP(0); Fl; 


Jump to leaf specific flow 


Flags Affected 
See individual leaf functions. 


Protected Mode Exceptions 


#UD If any of the LOCK/66H/REP/VEX prefixes are used. 
If current privilege level is not 0. 
If CPUID.(EAX=12H,ECX=0):EAX.OSS [bit 5] = 0. 
If logical processor is in SMM. 

#GP(0) If IA32_FEATURE_CONTROL.LOCK = 0. 


If IA32_FEATURE_CONTROL.SGX_ENABLE = 0. 

If input value in EAX encodes an unsupported leaf. 
If data segment expand down. 

If CRO.PG=0. 


Real-Address Mode Exceptions 


#UD 


ENCLV is not recognized in real mode. 


Virtual-8086 Mode Exceptions 


#UD 


ENCLV is not recognized in virtual-8086 mode. 


Compatibility Mode Exceptions 


Same exceptions as in protected mode. 


64-Bit Mode Exceptions 


#UD If any of the LOCK/66H/REP/VEX prefixes are used. 
If current privilege level is not 0. 
If CPUID.(EAX=12H,ECX=0):EAX.OSS [bit 5] = 0. 
If logical processor is in SMM. 

#GP(0) If IA32_FEATURE_CONTROL.LOCK = 0. 


If IA32_FEATURE_CONTROL.SGX_ENABLE = 0. 
If input value in EAX encodes an unsupported leaf. 
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36.3. INTEL® SGX SYSTEM LEAF FUNCTION REFERENCE 


Leaf functions available with the ENCLS instruction mnemonic are covered in this section. In general, each instruc- 
tion leaf requires EAX to specify the leaf function index and/or additional implicit registers specifying leaf-specific 
input parameters. An instruction operand encoding table provides details of each implicit register usage and asso- 
ciated input/output semantics. 

In many cases, an input parameter specifies an effective address associated with a memory object inside or outside 
the EPC, the memory addressing semantics of these memory objects are also summarized in a separate table. 


36-16 Vol. 3D 


INTEL® SGX INSTRUCTION REFERENCES 


EADD—Add a Page to an Uninitialized Enclave 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = 01H IR VIV SGX1 This leaf function adds a page to an uninitialized enclave. 
ENCLS[EADD] 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EADD (In) Address of a PAGEINFO (In) Address of the destination EPC page (In) 
Description 


This leaf function copies a source page from non-enclave memory into the EPC, associates the EPC page with an 
SECS page residing in the EPC, and stores the linear address and security attributes in EPCM. As part of the asso- 
ciation, the enclave offset and the security attributes are measured and extended into the SECS.MRENCLAVE. This 
instruction can only be executed when current privilege level is 0. 


RBX contains the effective address of a PAGEINFO structure while RCX contains the effective address of an EPC 
page. The table below provides additional information on the memory parameter of EADD leaf function. 


EADD Memory Parameter Semantics 


PAGEINFO PAGEINFO.SECS PAGEINFO.SRCPGE PAGEINFO.SECINFO EPCPAGE 
Read access permitted | Read/Write access permit- | Read access permitted | Read access permitted | Write access permitted 
by Non Enclave ted by Enclave by Non Enclave by Non Enclave by Enclave 


The instruction faults if any of the following: 


EADD Faulting Conditions 


The operands are not properly aligned. Unsupported security attributes are set. 

Refers to an invalid SECS. Reference is made to an SECS that is locked by another thread. 

The EPC page is locked by another thread. RCX does not contain an effective address of an EPC page. 

The EPC page is already valid. If security attributes specifies a TCS and the source page specifies unsupported 
TCS values or fields. 

The SECS has been initialized. The specified enclave offset is outside of the enclave address space. 


Concurrency Restrictions 


Table 36-8. Base Concurrency Restrictions of EADD 


Base Concurrency Restrictions 
Leaf Parameter - ; poe 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EADD Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
SECS [DS:RBX]PAGEINFO.SECS Shared #GP 
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Table 36-9. Additional Concurrency Restrictions of EADD 


Additional Concurrency Restrictions 
Leaf Parameter Gone Alans pales vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EADD Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGE- | Concurrent Exclusive #GP Concurrent 
INFO.SECS 
Operation 
Temp Variables in EADD Operational Flow 
Name Type Size (bits) | Description 
TMP_SRCPGE Effective Address 32/64 Effective address of the source page. 
TMP_SECS Effective Address 32/64 Effective address of the SECS destination page. 
TMP_SECINFO Effective Address 32/64 Effective address of an SECINFO structure which contains security 
attributes of the page to be added. 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:TMP_SECINFO. 
TMP_LINADDR Unsigned Integer 64 Holds the linear address to be stored in the EPCM and used to 
calculate TMP_ENCLAVEOFFSET. 
TMP_ENCLAVEOFFSET Enclave Offset 64 The page displacement from the enclave base address. 
TMPUPDATEFIELD SHA256 Buffer 512 Buffer used to hold data being added to TMP_SECS.MRENCLAVE. 


IF (DS:RBX is not 32Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


TMP_SRCPGE := DS:RBX.SRCPGE; 
TMP_SECS := DS:RBX.SECS; 
TMP_SECINFO := DS:RBX.SECINFO; 
TMP_LINADDR := DS:RBX.LINADDR; 


IF (DS:TMP_SRCPGE is not 4KByte aligned or DS:TMP_SECS is not 4KByte aligned or 
DS:TMP_SECINFO is not 64Byte aligned or TMP_LINADDR is not 4KByte aligned) 
THEN #GP(0); Fl; 


IF (DS: TMP_SECS does not resolve within an EPC) 
THEN #PF(DS:TMP_SECS); FI; 


SCRATCH_SECINFO := DS:TMP_SECINFO; 


(* Check for misconfigured SECINFO flags*) 
IF (SCRATCH_SECINFO reserved fields are not zero or 
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| (SCRATCH_SECINFO.FLAGS.PT is PT_REG or SCRATCH_SECINFO.FLAGSPT is PT_TCS or 
(SCRATCH_SECINFO.FLAGSPT is PT_SS_FIRST and CPUID.(EAX=12H, ECX=1):EAX[6] = 1) or 
(SCRATCH_SECINFO.FLAGSPT is PT_SS_REST and CPUID.(EAX=12H, ECX=1):EAX[6] = 1) ) 
THEN #GP(0); Fl; 


(* If PT_SS_FIRST/PT_SS_REST page types are requested then CR4.CET must be 1 *) 
IF ((SCRATCH_SECINFO.FLAGS.PT is PT_SS_FIRST OR 

SCRATCH_SECINFO.FLAGS.PT is PT_SS_REST) AND CR4.CET == 0) 

THEN #GP(0); Fl; 


(* Check the EPC page for concurrency *) 
IF (EPC page is not available for EADD) 


THEN 
IF (<<VMX non-root operation>> AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := << translation of DS:RCX produced by paging >>; 
VMCS.Guest-linear_address := DS:RCX; 
Deliver VMEXIT; 
ELSE 
#GP(0); 
Fl; 
Fl; 
IF (EPCM(DS:RCX).VALID # 0) 


THEN #PF(DS:RCX); Fl; 


(* Check the SECS for concurrency *) 
IF (SECS is not available for EADD) 
THEN #GP(0); Fl; 


IF (EPCM(DS:TMP_SECS).VALID = 0 or EPCM(DS:TMP_SECS).PT # PT_SECS) 
THEN #PF(DS:TMP_SECS); FI; 


(* Copy 4KBytes from source page to EPC page”) 
DS:RCX[32767:0] := DS:TMP_SRCPGE[32767:0]; 


CASE (SCRATCH_SECINFO.FLAGS.PT) 


PT_TCS: 
IF (DS:RCX.RESERVED # 0) #GP(0); FI; 
IF ((DS:TMP_SECS.ATTRIBUTES.MODE64BIT = 0) and 
((DS:TCS.FSLIMIT & OFFFH # OFFFH) or (DS:TCS.GSLIMIT & OFFFH # OFFFH) )) #GP(O); FI; 
(* Ensure TCS.PREVSSP is zero *) 
IF (CPUID.(EAX=07H, ECX=00h):ECX[CET_SS] = 1) and (DS:RCX.PREVSSP != 0) #GP(0); FI; 
BREAK; 
PT_REG: 
IF (SCRATCH_SECINFO.FLAGS.W = 1 and SCRATCH_SECINFO.FLAGS.R = 0) #GP(0); FI; 
BREAK; 
PT_SS_FIRST: 
PT_SS_REST: 
(* SS pages cannot created on first or last page of ELRANGE *) 
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IF (TMP_LINADDR = DS:TMP_SECS.BASEADDR or TMP_LINADDR = (DS:TMP_SECS.BASEADDR + DS:TMP_SECS.SIZE - 0x1000) ) 
THEN #GP(0); Fl; 
IF (DS:RCX[4087:0] != 0 ) #GP(0); Fl; 
IF (SCRATCH_SECINFO.FLAGS.PT == PT_SS_FIRST) 
THEN 
(* Check that valid RSTORSSP token exists *) 
IF (DS:RCX[4095:4088] != ((TMP_LINADDR + 0x1000) | DS:TMP_SECS.ATTRIBUTES.MODE64BIT) ) #GP(0); FI; 
(* Check the 8 bytes are zero *) 
IF ( DS:RCX[4095:4088] != 0 ) #GP(0); FI; 
Fl; 
IF (SCRATCH_SECINFO.FLAGS.W = 0 OR SCRATCH_SECINFO.FLAGS.R = 0 OR 
SCRATCH_SECINFO.FLAGS.X = 1) #GP(0); FI; 
BREAK; 
ESAC; 


(* Check the enclave offset is within the enclave linear address space *) 
IF (TMP_LINADDR < DS:TMP_SECS.BASEADDR or TMP_LINADDR 2 DS:TMP_SECS.BASEADDR + DS:TMP_SECS.SIZE) 
THEN #GP(0); Fl; 


(* Check concurrency of measurement resource”) 
IF (Measurement being updated) 
THEN #GP(0); Fl; 


(* Check if the enclave to which the page will be added is already in Initialized state *) 
IF (DS:TMP_SECS already initialized) 
THEN #GP(0); Fl; 


(* For TCS pages, force EPCM.rwx bits to 0 and no debug access *) 
IF (SCRATCH_SECINFO.FLAGS.PT = PT_TCS) 
THEN 

SCRATCH_SECINFO.FLAGS.R := 0; 
SCRATCH_SECINFO.FLAGS.W := 0; 
SCRATCH_SECINFO.FLAGS.X := 0; 
(DS:RCX).FLAGS.DBGOPTIN := 0; // force TCS.FLAGS.DBGOPTIN off 
DS:RCX.CSSA := 0; 
DS:RCX.AEP := 0; 
DS:RCX.STATE := 0; 

Fl; 


(* Add enclave offset and security attributes to MRENCLAVE *) 

TMP_ENCLAVEOFFSET := TMP_LINADDR - DS:TMP_SECS.BASEADDR; 

TMPUPDATEFIELD[63:0] := 0000000044444145H; // “EADD" 

TMPUPDATEFIELD[1 27:64] := TMP_ENCLAVEOFFSET; 

TMPUPDATEFIELD[51 1:128] := SCRATCH_SECINFO[375:0]; // 48 bytes 
DS:TMP_SECS.MRENCLAVE := SHA256UPDATE(DS:TMP_SECS.MRENCLAVE, TMPUPDATEFIELD) 
INC enclave's MRENCLAVE update counter; 


(* Add enclave offset and security attributes to MRENCLAVE *) 
EPCM(DS:RCX).R := SCRATCH_SECINFO.FLAGS.R; 
EPCM(DS:RCX).W := SCRATCH_SECINFO.FLAGS.W; 
EPCM(DS:RCX).X := SCRATCH_SECINFO.FLAGS.X; 
EPCM(DS:RCX).PT := SCRATCH_SECINFO.FLAGS.PT; 
EPCM(DS:RCX).ENCLAVEADDRESS := TMP_LINADDR; 


36-20 Vol. 3D 


INTEL® SGX INSTRUCTION REFERENCES 


(* associate the EPCPAGE with the SECS by storing the SECS identifier of DS:TMP_SECS *) 
Update EPCM(DS:RCX) SECS identifier to reference DS:TMP_SECS identifier; 


(* Set EPCM entry fields *) 
EPCM(DS:RCX).BLOCKED := 0; 
EPCM(DS:RCX).PENDING := 0; 
EPCM(DS:RCX).MODIFIED := 0; 
EPCM(DS:RCX).VALID := 1; 


Flags Affected 
None 


Protected Mode Exceptions 
#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If an enclave memory operand is outside of the EPC. 
If an enclave memory operand is the wrong type. 
If a memory operand is locked. 
If the enclave is initialized. 
If the enclave's MRENCLAVE is locked. 
If the TCS page reserved bits are set. 
If the TCS page PREVSSP field is not zero. 
If the PT_SS_REST or PT_SS_REST page is the first or last page in the enclave. 
If the PT_SS_FIRST or PT_SS_REST page is not initialized correctly. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If the EPC page is valid. 


64-Bit Mode Exceptions 
#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If an enclave memory operand is outside of the EPC. 
If an enclave memory operand is the wrong type. 
If a memory operand is locked. 
If the enclave is initialized. 
If the enclave's MRENCLAVE is locked. 
If the TCS page reserved bits are set. 
If the TCS page PREVSSP field is not zero. 
If the PT_SS_REST or PT_SS_REST page is the first or last page in the enclave. 
If the PT_SS_FIRST or PT_SS_REST page is not initialized correctly. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If the EPC page is valid. 
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EAUG—Add a Page to an Initialized Enclave 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = ODH IR VV SGX2 This leaf function adds a page to an initialized enclave. 
ENCLS[EAUG] 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EAUG (In) Address of a SECINFO (In) Address of the destination EPC page (In) 
Description 


This leaf function zeroes a page of EPC memory, associates the EPC page with an SECS page residing in the EPC, 
and stores the linear address and security attributes in the EPCM. As part of the association, the security attributes 
are configured to prevent access to the EPC page until a corresponding invocation of the EACCEPT leaf or EACCEPT- 
COPY leaf confirms the addition of the new page into the enclave. This instruction can only be executed when 
current privilege level is 0. 


RBX contains the effective address of a PAGEINFO structure while RCX contains the effective address of an EPC 
page. The table below provides additional information on the memory parameter of the EAUG leaf function. 


EAUG Memory Parameter Semantics 
PAGEINFO PAGEINFO.SECS PAGEINFO.SRCPGE PAGEINFO.SECINFO EPCPAGE 


Read access permit- | Read/Write access permit- Read access permitted by | Write access permitted by 
Must be zero 
ted by Non Enclave ted by Enclave Non Enclave Enclave 


The instruction faults if any of the following: 


EAUG Faulting Conditions 


The operands are not properly aligned. Unsupported security attributes are set. 

Refers to an invalid SECS. Reference is made to an SECS that is locked by another thread. 

The EPC page is locked by another thread. RCX does not contain an effective address of an EPC page. 

The EPC page is already valid. The specified enclave offset is outside of the enclave address space. 
The SECS has been initialized. 


Concurrency Restrictions 


Table 36-10. Base Concurrency Restrictions of EAUG 


Base Concurrency Restrictions 
Leaf Parameter - i oe 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EAUG Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
SECS [DS:RBX]PAGEINFO.SECS Shared #GP 
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Additional Concurrency Restrictions 
Leaf Parameter OEE Aleta peat vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EAUG Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGE- | Concurrent Concurrent Concurrent 
INFO.SECS 
Operation 
Temp Variables in EAUG Operational Flow 
Name Type Size (bits) | Description 
TMP_SECS Effective Address 32/64 Effective address of the SECS destination page. 
TMP_SECINFO Effective Address 32/64 Effective address of an SECINFO structure which contains security 
attributes of the page to be added. 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:TMP_SECINFO. 
TMP_LINADDR Unsigned Integer 64 Holds the linear address to be stored in the EPCM and used to 
calculate TMP_ENCLAVEOFFSET. 


IF (DS:RBX is not 32Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


TMP_SECS := DS:RBX.SECS; 
TMP_SECINFO := DS:RBX.SECINFO; 
IF (DS:RBX.SECINFO is not 0) 
THEN 
IF (DS:TMP_SECINFO is not 648B aligned) 
THEN #GP(0); Fl; 
Fl; 


TMP_LINADDR := DS:RBX.LINADDR; 


IF (DS:TMP_SECS is not 4KByte aligned or TMP_LINADDR is not 4KByte aligned ) 
THEN #GP(0); Fl; 


IF DS:RBX.SRCPAGE is not 0 
THEN #GP(0); Fl; 


IF (DS:TMP_SECS does not resolve within an EPC) 
THEN #PF(DS:TMP_SECS); FI; 


(* Check the EPC page for concurrency *) 
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IF (EPC page in use) 


THEN 
IF (<<VMX non-root operation> > AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := << translation of DS:RCX produced by paging >>; 
VMCS.Guest-linear_address := DS:RCX; 
Deliver VMEXIT; 
ELSE 
#GP(0); 
Fl; 
Fl: 
IF (EPCM(DS:RCX).VALID # 0) 


THEN #PF(DS:RCX); Fl; 


(* copy SECINFO contents into a scratch SECINFO *) 
IF (DS:RBX.SECINFO is 0) 
THEN 
(* allocate and initialize a new scratch SECINFO structure *) 
SCRATCH_SECINFO.PT := PT_REG; 
SCRATCH_SECINFO.LR := 1; 
SCRATCH_SECINFO.W := 1; 
SCRATCH_SECINFO.X := 0; 
<< zero out remaining fields of SCRATCH_SECINFO >> 
ELSE 
(* copy SECINFO contents into scratch SECINFO *) 
SCRATCH_SECINFO := DS:TMP_SECINFO; 
(* check SECINFO flags for misconfiguration *) 
(* reserved flags must be zero *) 
(* SECINFO.FLAGS.PT must either be PT_SS_FIRST, or PT_SS_REST *) 
IF ((SCRATCH_SECINFO reserved fields are not O) or 
CPUID.(EAX=12H, ECX=1):EAX[6] is 0) OR 
(SCRATCH_SECINFO.PT is not PT_SS_FIRST, or PT_SS_REST) OR 
( (SCRATCH_SECINFO.FLAGS.R is 0) OR (SCRATCH_SECINFO.FLAGS.W is 0) OR (SCRATCH_SECINFO.FLAGS.X is 1) ) ) 
THEN #GP(0); Fl; 
FI; 
(* Check if PT_SS_FIRST/PT_SS_REST page types are requested then CR4.CET must be 1 *) 
IF ((SCRATCH_SECINFO.PT is PT_SS_FIRST OR SCRATCH_SECINFO.PT is PT_SS_REST) AND CR4.CET == 0) 
THEN #GP(0); Fl; 


(* Check the SECS for concurrency *) 
IF (SECS is not available for EAUG) 
THEN #GP(0); Fl; 


IF (EPCM(DS:TMP_SECS).VALID = 0 or EPCM(DS:TMP_SECS).PT # PT_SECS) 
THEN #PF(DS:TMP_SECS); FI; 


(* Check if the enclave to which the page will be added is in the Initialized state *) 


IF (DS:TMP_SECS is not initialized) 
THEN #GP(0); Fl; 
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(* Check the enclave offset is within the enclave linear address space *) 
IF ((TMP_LINADDR < DS:TMP_SECS.BASEADDR) or (TMP_LINADDR 2 DS:TMP_SECS.BASEADDR + DS:TMP_SECS.SIZE) ) 
THEN #GP(0); Fl; 


IF ((SCRATCH_SECINFO.PT is PT_SS_FIRST OR SCRATCH_SECINFO.PT is PT_SS_REST) ) 
THEN 

(* SS pages cannot created on first or last page of ELRANGE *) 

IF (TMP_LINADDR == DS:TMP_SECS.BASEADDR OR 

TMP_LINADDR == (DS:TMP_SECS.BASEADDR + DS:TMP_SECS.SIZE - 0x1000) ) 

THEN 
#GP(0); Fl; 

Fl; 


(* Clear the content of EPC page*) 
DS:RCX[32767:0] := 0; 


(* Set EPCM security attributes *) 

EPCM(DS:RCX).R := SCRATCH_SECINFO.FLAGS.R; 
EPCM(DS:RCX).W := SCRATCH_SECINFO.FLAGS.W; 
EPCM(DS:RCX).X := SCRATCH_SECINFO.FLAGS.X; 
EPCM(DS:RCX).PT := SCRATCH_SECINFO.FLAGS.PT; 
EPCM(DS:RCX).ENCLAVEADDRESS := TMP_LINADDR; 
EPCM(DS:RCX).BLOCKED := 0; 
EPCM(DS:RCX).PENDING := 1; 
EPCM(DS:RCX).MODIFIED := 0; 

EPCM(DS:RCX).PR := 0; 


SOOO OOOO 


(* associate the EPCPAGE with the SECS by storing the SECS identifier of DS:TMP_SECS *) 
Update EPCM(DS:RCX) SECS identifier to reference DS:TMP_SECS identifier; 


(* Set EPCM valid fields *) 
EPCM(DS:RCX).VALID := 1; 


Flags Affected 
None 


Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 
If the enclave is not initialized. 

#PF(error code) If a page fault occurs in accessing memory operands. 


64-Bit Mode Exceptions 
#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 
If the enclave is not initialized. 
#PF(error code) If a page fault occurs in accessing memory operands. 
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EBLOCK—Mark a page in EPC as Blocked 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = OSH IR VV SGX1 This leaf function marks a page in the EPC as blocked. 
ENCLS[EBLOCK] 
Instruction Operand Encoding 
Op/En EAX RCX 
IR EBLOCK (In) Return error code (Out) Effective address of the EPC page (In) 
Description 


This leaf function causes an EPC page to be marked as BLOCKED. This instruction can only be executed when 
current privilege level is 0. 


The content of RCX is an effective address of an EPC page. The DS segment is used to create linear address. 
Segment override is not supported. 


An error code is returned in RAX. 


The table below provides additional information on the memory parameter of EBLOCK leaf function. 


EBLOCK Memory Parameter Semantics 


EPCPAGE 


Read/Write access permitted by Enclave 


The error codes are: 


Table 36-12. EBLOCK Return Value in RAX 


No Error 


Error Code (see Table 36-4) 


EBLOCK successful. 


Description 


SGX_BLKSTATE 


Page already blocked. This value is used to indicate to a VMM that the page was already in 
BLOCKED state as a result of EBLOCK and thus will need to be restored to this state when it is 
eventually reloaded (using ELDB). 


SGX_ENTRYEPOCH_LOCKED 


SECS locked for Entry Epoch update. This value indicates that an ETRACK is currently 
executing on the SECS. The EBLOCK should be reattempted. 


SGX_NOTBLOCKABLE 


Page type is not one which can be blocked. 


SGX_PG_INVLD 


SGX_EPC_PAGE_CONFLICT 


Page is not valid and cannot be blocked. 
Page is being written by EADD, EAUG, ECREATE, ELDU/B, EMODT, or EWB. 


Concurrency Restrictions 


Table 36-13. Base Concurrency Restrictions of EBLOCK 
Base Concurrency Restrictions 


Leaf Parameter : ; Teer 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
EBLOCK Target [DS:RCX] Shared SGX_EPC_PAGE_ 
CONFLICT 
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Table 36-14. Additional Concurrency Restrictions of EBLOCK 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EBLOCK Target [DS:RCX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EBLOCK Operational Flow 
Name Type Size (Bits) Description 
TMP_BLKSTATE Integer 64 Page is already blocked. 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


RFLAGS.ZF,CF,PF,AF,OF,SF := 0; 
RAX := 0; 


(* Check the EPC page for concurrency”) 
IF (EPC page in use) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO DONE; 
Fl; 


IF (EPCM(DS:RCX). VALID = 0) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_PG_INVLD; 
GOTO DONE; 
Fl; 


IF ((EPCM(DS:RCX).PT # PT_REG) and (EPCM(DS:RCX).PT # PT_TCS) and (EPCM(DS:RCX).PT # PT_TRIM) 
and EPCM(DS:RCX).PT # PT_SS_FIRST) and (EPCM(DS:RCX).PT # PT_SS_REST)) 
THEN 
RFLAGS.CF := 1; 
IF (EPCM(DS:RCX).PT = PT_SECS) 
THEN RAX := SGX_PG_IS_SECS; 
ELSE RAX := SGX_NOTBLOCKABLE; 
Fl; 
GOTO DONE; 
Fl; 


(* Check if the page is already blocked and report blocked state *) 
TMP_BLKSTATE := EPCM(DS:RCX).BLOCKED; 
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(* at this point, the page must be valid and PT_TCS or PT_REG or PT_TRIM*) 
IF (TMP_BLKSTATE = 1) 
THEN 
RFLAGS.CF := 1; 
RAX := SGX_BLKSTATE; 
ELSE 
EPCM(DS:RCX).BLOCKED := 1 
Fl; 
DONE: 


Flags Affected 


Sets ZF if SECS is in use or invalid, otherwise cleared. Sets CF if page is BLOCKED or not blockable, otherwise 
cleared. Clears PF, AF, OF, SF. 


Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If the specified EPC resource is in use. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 


64-Bit Mode Exceptions 

#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If the specified EPC resource is in use. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 
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ECREATE—Create an SECS page in the Enclave Page Cache 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = OOH IR VV SGX1 This leaf function begins an enclave build by creating an SECS 
ENCLS[ECREATE] page in EPC. 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR ECREATE (In) Address of a PAGEINFO (In) Address of the destination SECS page (In) 
Description 


ENCLS[ECREATE] is the first instruction executed in the enclave build process. ECREATE copies an SECS structure 
outside the EPC into an SECS page inside the EPC. The internal structure of SECS is not accessible to software. 


ECREATE will set up fields in the protected SECS and mark the page as valid inside the EPC. ECREATE initializes or 
checks unused fields. 


Software sets the following fields in the source structure: SECS:BASEADDR, SECS:SIZE in bytes, ATTRIBUTES, 
CONFIGID and CONFIGSVN. SECS:BASEADDR must be naturally aligned on an SECS.SIZE boundary. SECS.SIZE 
must be at least 2 pages (8192). 


The source operand RBX contains an effective address of a PAGEINFO structure. PAGEINFO contains an effective 
address of a source SECS and an effective address of an SECINFO. The SECS field in PAGEINFO is not used. 


The RCX register is the effective address of the destination SECS. It is an address of an empty slot in the EPC. The 
SECS structure must be page aligned. SECINFO flags must specify the page as an SECS page. 


ECREATE Memory Parameter Semantics 


PAGEINFO PAGEINFO.SRCPGE PAGEINFO.SECINFO EPCPAGE 
Read access permitted by | Read access permitted by Read access permitted by Non Write access permitted by 
Non Enclave Non Enclave Enclave Enclave 


ECREATE will fault if the SECS target page is in use; already valid; outside the EPC. It will also fault if addresses are 
not aligned; unused PAGEINFO fields are not zero. 


If the amount of space needed to store the SSA frame is greater than the amount specified in SECS.SSAFRAME- 
SIZE, a #GP(0) results. The amount of space needed for an SSA frame is computed based on 
DS:TMP_SECS.ATTRIBUTES.XFRM size. Details of computing the size can be found Section 37.7. 


Concurrency Restrictions 


Table 36-15. Base Concurrency Restrictions of ECREATE 


Base Concurrency Restrictions 
Leaf Parameter - - —— 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
ECREATE SECS [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
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Table 36-16. Additional Concurrency Restrictions of ECREATE 


Additional Concurrency Restrictions 
Leaf Parameter Gone Alain ane vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
ECREATE | SECS [DS:RCX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in ECREATE Operational Flow 
Name Type Size (Bits) | Description 
TMP_SRCPGE Effective Address 32/64 Effective address of the SECS source page. 
TMP_SECS Effective Address 32/64 Effective address of the SECS destination page. 
TMP_SECINFO Effective Address 32/64 Effective address of an SECINFO structure which contains security 
attributes of the SECS page to be added. 
TMP_XSIZE SSA Size 64 The size calculation of SSA frame. 
TMP_MISC_SIZE MISC Field Size 64 Size of the selected MISC field components. 
TMPUPDATEFIELD SHA256 Buffer 512 Buffer used to hold data being added to TMP_SECS.MRENCLAVE. 


IF (DS:RBX is not 32Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


TMP_SRCPGE := DS:RBX.SRCPGE; 
TMP_SECINFO := DS:RBX.SECINFO; 


IF (DS: TMP_SRCPGE is not 4KByte aligned or DS:TMP_SECINFO is not 64Byte aligned) 
THEN #GP(0); Fl; 


IF (DS:RBX.LINADDR ! = 0 or DS:RBX.SECS # 0) 
THEN #GP(0); Fl; 


(* Check for misconfigured SECINFO flags*) 
IF (DS:TMP_SECINFO reserved fields are not zero or DS:TMP_SECINFO.FLAGS.PT # PT_SECS) 
THEN #GP(0); Fl; 


TMP_SECS := RCX; 


IF (EPC entry in use) 
THEN 
IF (<<VMX non-root operation> > AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 


36-30 Vol. 3D 


INTEL® SGX INSTRUCTION REFERENCES 


VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := 
<< translation of DS:TMP_SECS produced by paging >>; 

VMCS.Guest-linear_address := DS:TMP_SECS; 

Deliver VMEXIT; 

ELSE 
#GP(0); 

Fl; 
Fl; 


IF (EPC entry in use) 
THEN #GP(0); Fl; 


IF (EPCM(DS:RCX).VALID = 1) 
THEN #PF(DS:RCX); Fl; 


(* Copy 4KBytes from source page to EPC page”) 
DS:RCX[32767:0] := DS:TMP_SRCPGE[32767:0]; 


(* Check lower 2 bits of XFRM are set *) 
IF (( DS:TMP_SECS.ATTRIBUTES.XFRM BitwiseAND 03H) # 03H) 
THEN #GP(0); Fl; 


IF (XFRM is illegal) 
THEN #GP(0); Fl; 


(* Check legality of CET_ATTRIBUTES *) 

IF ((DS:TMP_SECS.ATTRIBUTES.CET = 0 and DS:TMP_SECS.CET_ATTRIBUTES # 0) || 
(DS:TMP_SECS.ATTRIBUTES.CET = 0 and DS:TMP_SECS.CET_LEG_BITMAP_OFFSET # 0) || 
(CPUID.(EAX=7, ECX=0):EDX[CET_IBT] = 0 and DS:TMP_SECS.CET_LEG_BITMAP_OFFSET # 0) || 
(CPUID.(EAX=7, ECX=0):EDX[CET_IBT] = 0 and DS:TMP_SECS.CET_ATTRIBUTES[5:2] # 0) || 
(CPUID.(EAX=7, ECX=0):ECX[CET_SS] = 0 and DS:TMP_SECS.CET_ATTRIBUTES[1:0] # 0) || 
(DS:TMP_SECS.ATTRIBUTES.MODEG4BIT = 1 and 
(DS:TMP_SECS.BASEADDR + DS:TMP_SECS.CET_LEG_BITMAP_OFFSET) not canonical) || 
(DS:TMP_SECS.ATTRIBUTES.MODEG4BIT = 0 and 
(DS:TMP_SECS.BASEADDR + DS:TMP_SECS.CET_LEG_BITMAP_OFFSET) & OxFFFFFFFFOOO00000) || 
(DS:TMP_SECS.CET_ATTRIBUTES.reserved fields not 0) or 
(DS:TMP_SECS.CET_LEG_BITMAP_OFFSET) is not page aligned)) 

THEN 
#GP(0); 


Fl; 


(* Make sure that the SECS does not have any unsupported MISCSELECT options*) 
IF ( (CPUID.(EAX=1 2H, ECX=0):EBX[31:0] & DS:TMP_SECS.MISCSELECT[31:0)) ) 
THEN 
EPCM(DS:TMP_SECS).EntryLock.Release(); 
#GP(0); 
Fl; 


( * Compute size of MISC area *) 
TMP_MISC_SIZE := compute_misc_region_size(); 


(* Compute the size required to save state of the enclave on async exit, see Section 37.7.2.2*) 
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TMP_XSIZE := compute_xsave_size(DS:TMP_SECS.ATTRIBUTES.XFRM) + GPR_SIZE + TMP_MISC_SIZE; 


(* Ensure that the declared area is large enough to hold XSAVE and GPR stat *) 
IF (DS:TMP_SECS.SSAFRAMESIZE*4096 < TMP_XSIZE) 
THEN #GP(0); Fl; 


IF ( (DS:TMP_SECS.ATTRIBUTES.MODE64BIT = 1) and (DS:TMP_SECS.BASEADDR is not canonical) ) 
THEN #GP(0); Fl; 


IF ( (DS:TMP_SECS.ATTRIBUTES.MODE64BIT = 0) and (DS:TMP_SECS.BASEADDR and OFFFFFFFFOOO00000H) ) 
THEN #GP(0); Fl; 


IF ( (DS:TMP_SECS.ATTRIBUTES.MODE64BIT = 0) and (DS:TMP_SECS.SIZE 2 2 * (CPUID.(EAX=12H, ECX=0):.EDX[7:0]) ) ) 
THEN #GP(0); Fl; 


IF ( (DS:TMP_SECS.ATTRIBUTES.MODE64BIT = 1) and (DS:TMP_SECS.SIZE > 2 * (CPUID.(EAX=12H, ECX=0):.EDX[15:8}) ) ) 
THEN #GP(0); Fl; 


(* Enclave size must be at least 8192 bytes and must be power of 2 in bytes*) 
IF (DS:TMP_SECS.SIZE < 8192 or popcnt(DS:TMP_SECS.SIZE) > 1) 
THEN #GP(0); Fl; 


(* Ensure base address of an enclave is aligned on size*) 
IF ( (DS:TMP_SECS.BASEADDR and (DS:TMP_SECS.SIZE-1) ) ) 
THEN #GP(0); Fl; 


(* Ensure the SECS does not have any unsupported attributes*) 
IF (DS: TMP_SECS.ATTRIBUTES and (~CR_SGX_ATTRIBUTES_MASK) ) 
THEN #GP(0); Fl; 


IF (DS:TMP_SECS reserved fields are not zero) 
THEN #GP(0); Fl; 


(* Verify that CONFIGID/CONFIGSVN are not set with attribute *) 
IF (((DS:TMP_SECS.CONFIGID # 0) or (DS:TMP_SECS.CONFIGSVN #0)) AND (DS:TMP_SECS.ATTRIBUTES.KSS == 0 )) 
THEN #GP(0); Fl; 


Clear DS:TMP_SECS to Uninitialized; 

DS:TMP_SECS.MRENCLAVE := SHA256INITIALIZE(DS:TMP_SECS.MRENCLAVE); 
DS:TMP_SECS.ISVSVN := 0; 

DS:TMP_SECS.ISVPRODID := 0; 


(* Initialize hash updates etc*) 
Initialize enclave’s MRENCLAVE update counter; 


(* Add “ECREATE” string and SECS fields to MRENCLAVE *) 
TMPUPDATEFIELD[63:0] := 0045544145524345H; // “ECREATE" 
TMPUPDATEFIELD[95:64] := DS:TMP_SECS.SSAFRAMESIZE; 
TMPUPDATEFIELD[1 59:96] := DS:TMP_SECS.SIZE; 
IF (CPUID.(EAX=7, ECX=0):EDX[CET_IBT] = 1) 
THEN 
TMPUPDATEFIELD[223:1 60] := DS:TMP_SECS.CET_LEG_BITMAP_OFFSET; 
ELSE 
TMPUPDATEFIELD[223:160] := 0; 


36-32 Vol. 3D 


INTEL® SGX INSTRUCTION REFERENCES 


Fl; 

TMPUPDATEFIELD[51 1:160] := 0; 

DS:TMP_SECS.MRENCLAVE := SHA256UPDATE(DS:TMP_SECS.MRENCLAVE, TMPUPDATEFIELD) 
INC enclave’s MRENCLAVE update counter; 


(* Set EID *) 
DS:TMP_SECS.EID := LockedXAdd(CR_NEXT_EID, 1); 


(* Initialize the virtual child count to zero *) 
DS:TMP_SECS.VIRTCHILDCNT := 0; 


(* Load ENCLAVECONTEXT with Address out of paging of SECS *) 
<< store translation of DS:RCX produced by paging in SECS(DS:RCX).ENCLAVECONTEXT >> 


(* Set the EPCM entry, first create SECS identifier and store the identifier in EPCM *) 
EPCM(DS:TMP_SECS).PT := PT_SECS; 

EPCM(DS:TMP_SECS).ENCLAVEADDRESS := 0; 

EPCM(DS:TMP_SECS).R := 0; 

EPCM(DS:TMP_SECS).W := 0; 

EPCM(DS:TMP_SECS).X := 0; 


(* Set EPCM entry fields *) 

EPCM(DS:RCX).BLOCKED := 0; 
EPCM(DS:RCX).PENDING := 0; 
EPCM(DS:RCX).MODIFIED := 0; 
EPCM(DS:RCX).PR := 0; 
EPCM(DS:RCX).VALID := 1; 


Flags Affected 
None 


Protected Mode Exceptions 


#GP(0) If a memory operand effective address is outside the DS segment limit. 


If a memory operand is not properly aligned. 
If the reserved fields are not zero. 
If PAGEINFO.SECS is not zero. 
If PAGEINFO.LINADDR is not zero. 
If the SECS destination is locked. 
If SECS.SSAFRAMESIZE is insufficient. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If the SECS destination is outside the EPC. 


64-Bit Mode Exceptions 

#GP(0) If a memory address is non-canonical form. 
If a memory operand is not properly aligned. 
If the reserved fields are not zero. 
If PAGEINFO.SECS is not zero. 
If PAGEINFO.LINADDR is not zero. 
If the SECS destination is locked. 
If SECS.SSAFRAMESIZE is insufficient. 
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#PF(error code) If a page fault occurs in accessing memory operands. 
If the SECS destination is outside the EPC. 
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Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = 04H IR VV SGX1 This leaf function reads a dword/quadword from a debug enclave. 
ENCLS[EDBGRD] 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
IR EDBGRD (In) a Data read from a debug enclave (Out) | Address of source memory in the EPC (In) 
Description 


This leaf function copies a quadword/doubleword from an EPC page belonging to a debug enclave into the RBX 
register. Eight bytes are read in 64-bit mode, four bytes are read in non-64-bit modes. The size of data read cannot 


be overridden. 


The effective address of the source location inside the EPC is provided in the register RCX. 


EDBGRD Memory Parameter Semantics 


EPCQW 


Read access permitted by Enclave 


The error codes are: 


Table 36-17. EDBGRD Return Value in RAX 


Error Code (see Table 36-4) 


Description 


No Error 


EDBGRD successful. 


SGX_PAGE_NOT_DEBUGGABLE 


The EPC page cannot be accessed because it is in the PENDING or MODIFIED state. 


The instruction faults if any of the following: 


EDBGRD Faulting Conditions 


RCX points into a page that is an SECS. 


RCX does not resolve to a naturally aligned linear address. 


RCX points to a page that does not belong to an 
enclave that is in debug mode. 


RCX points to a location inside a TCS that is beyond the architectural size of the 
TCS (SGX_TCS_LIMIT). 


An operand causing any segment violation. 


May page fault. 


CPL > 0. 


This instruction ignores the EPCM RWX attributes on the enclave page. Consequently, violation of EPCM RWX attri- 
butes via EDBGRD does not result in a #GP. 
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Concurrency Restrictions 


Table 36-18. Base Concurrency Restrictions of EDBGRD 


Base Concurrency Restrictions 
Leaf Parameter 


Access On Conflict SGX_CONFLICT VM Exit Qualification 


EDBGRD Target [DS:RCX] Shared #GP 


Table 36-19. Additional Concurrency Restrictions of EDBGRD 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EDBGRD _ | Target [DS:RCX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EDBGRD Operational Flow 
Name Type Size (Bits) | Description 
TMP_MODE64 Binary 1 ((IA32_EFER.LMA = 1) && (CS.L = 1)) 
TMP_SECS 64 Physical address of SECS of the enclave to which source operand belongs. 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1); 


IF ((TMP_MODE64 = 1) and (DS:RCX is not SByte Aligned) ) 
THEN #GP(0); Fl; 


IF ( (TMP_MODE64 = 0) and (DS:RCX is not 4Byte Aligned) ) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* make sure no other Intel SGX instruction is accessing the same EPCM entry *) 
IF (Another instruction modifying the same EPCM entry is executing) 
THEN #GP(0); Fl; 


IF (EPCM(DS:RCX).VALID = 0) 
THEN #PF(DS:RCX); Fl; 


(* make sure that DS:RCX (SOURCE) is pointing to a PT_REG or PT_TCS or PT_VA or PT_SS_FIRST or PT_SS_REST *) 
IF ( (EPCM(DS:RCX).PT # PT_REG) and (EPCM(DS:RCX).PT # PT_TCS) and (EPCM(DS:RCX).PT # PT_VA) 
and (EPCM(DS:RCX).PT # PT_SS_FIRST) and (EPCM(DS:RCX).PT # PT_SS_REST)) 

THEN #PF(DS:RCX); Fl; 


(* make sure that DS:RCX points to an accessible EPC page *) 
IF (EPCM(DS:RCX).PENDING is not 0 or (EPCM(DS:RCX).MODIFIED is not 0) ) 
THEN 
RFLAGS.ZF := 1; 
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RAX := SGX_PAGE_NOT_DEBUGGABLE; 
GOTO DONE; 
Fl; 


(* If source is a TCS, then make sure that the offset into the page is not beyond the TCS size*) 
IF (( EPCM(DS:RCX). PT = PT_TCS) and ((DS:RCX) & FFFH 2 SGX_TCS_LIMIT) ) 
THEN #GP(0); Fl; 


(* make sure the enclave owning the PT_REG or PT_TCS page allow debug *) 
IF ((EPCM(DS:RCX).PT = PT_REG) or (EPCM(DS:RCX).PT = PT_TCS) ) 
THEN 
TMP_SECS := GET_SECS_ADDRESS; 
IF (TMP_SECS.ATTRIBUTES.DEBUG = 0) 
THEN #GP(0); Fl; 
IF ((TMP_MODE64 = 1) ) 
THEN RBX[63:0] := (DS:RCX)[63:0]; 
ELSE EBX[31:0] := (DS:RCX)[3 1:0]; 
Fl; 
ELSE 
TMP_64BIT_VAL[63:0] := (DS:RCX)[63:0] & (~O7H); // Read contents from VA slot 
IF (TMP_MODE64 = 1) 
THEN 
IF (TMP_64BIT_VAL # OH) 
THEN RBX[63:0] := OFFFFFFFFFFFFFFFFH; 
ELSE RBX[63:0] := OH; 
Fl; 
ELSE 
IF (TMP_64BIT_VAL # OH) 
THEN EBX[31:0] := OFFFFFFFFH; 
ELSE EBX[31:0] := OH; 
Fl; 
Fl; 


(* clear EAX and ZF to indicate successful completion *) 
RAX := 0; 
RFLAGS.ZF := 0; 


DONE: 
(* clear flags *) 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 


ZF is set if the page is MODIFIED or PENDING; RAX contains the error code. Otherwise ZF is cleared and RAX is set 
to 0. CF, PF, AF, OF, SF are cleared. 


Protected Mode Exceptions 
#GP(0) If the address in RCS violates DS limit or access rights. 
If DS segment is unusable. 
If RCX points to a memory location not 4Byte-aligned. 
If the address in RCX points to a page belonging to a non-debug enclave. 
If the address in RCX points to a page which is not PT_TCS, PT_REG or PT_VA. 
If the address in RCX points to a location inside TCS that is beyond SGX_TCS_LIMIT. 
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#PF(error code) If a page fault occurs in accessing memory operands. 
If the address in RCX points to a non-EPC page. 
If the address in RCX points to an invalid EPC page. 


64-Bit Mode Exceptions 
#GP(0) If RCX is non-canonical form. 

If RCX points to a memory location not 8Byte-aligned. 

If the address in RCX points to a page belonging to a non-debug enclave. 

If the address in RCX points to a page which is not PT_TCS, PT_REG or PT_VA. 

If the address in RCX points to a location inside TCS that is beyond SGX_TCS_LIMIT. 
#PF(error code) If a page fault occurs in accessing memory operands. 

If the address in RCX points to a non-EPC page. 

If the address in RCX points to an invalid EPC page. 
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Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = O5H IR VIV SGX1 This leaf function writes a dword/quadword to a debug enclave. 
ENCLS[EDBGWR] 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
IR EDBGWR (In) con Data to be written to a debug enclave (In) | Address of Target memory in the EPC (In) 
Description 


This leaf function copies the content in EBX/RBX to an EPC page belonging to a debug enclave. Eight bytes are 
written in 64-bit mode, four bytes are written in non-64-bit modes. The size of data cannot be overridden. 


The effective address of the target location inside the EPC is provided in the register RCX. 


EDBGWR Memory Parameter Semantics 


Write access permitted by Enclave 


EPCQW 


The instruction faults if any of the following: 


EDBGWR Faulting Conditions 


RCX points into a page that is an SECS. 


RCX does not resolve to a naturally aligned linear address. 


enclave that is in debug mode. 


RCX points to a page that does not belong to an 


An operand causing any segment violation. 


RCX points to a location inside a TCS that is not the FLAGS word. 


May page fault. 


CPL > 0. 


The error codes are: 


Error Code (see Table 36-4) 


Table 36-20. EDBGWR Return Value in RAX 


Description 


No Error 


EDBGWR successful. 


SGX_PAGE_NOT_DEBUGGABLE 


The EPC page cannot be accessed because it is in the PENDING or MODIFIED state. 


This instruction ignores the EPCM RWX attributes on the enclave page. Consequently, violation of EPCM RWX attri- 
butes via EDBGRD does not result in a #GP. 
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Concurrency Restrictions 


Table 36-21. Base Concurrency Restrictions of EDBGWR 


Base Concurrency Restrictions 
Leaf Parameter 


Access On Conflict SGX_CONFLICT VM Exit Qualification 


EDBGWR Target [DS:RCX] Shared #GP 


Table 36-22. Additional Concurrency Restrictions of EDBGWR 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EDBGWR | Target [DS:RCX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EDBGWR Operational Flow 
Name Type Size (Bits) | Description 
TMP_MODE64 Binary 1 ((IA32_EFER.LMA = 1) && (CS.L = 1)). 
TMP_SECS 64 Physical address of SECS of the enclave to which source operand belongs. 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1); 


IF ( (TMP_MODE64 = 1) and (DS:RCX is not SByte Aligned) ) 
THEN #GP(0); Fl; 


IF ( (TMP_MODE64 = 0) and (DS:RCX is not 4Byte Aligned) ) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* make sure no other Intel SGX instruction is accessing the same EPCM entry *) 
IF (Another instruction modifying the same EPCM entry is executing) 
THEN #GP(0); Fl; 


IF (EPCM(DS:RCX).VALID = 0) 
THEN #PF(DS:RCX); Fl; 


(* make sure that DS:RCX (DST) is pointing to a PT_REG or PT_TCS or PT_SS_FIRST or PT_SS_REST *) 
IF ((EPCM(DS:RCX).PT # PT_REG) and (EPCM(DS:RCX).PT # PT_TCS) 
and (EPCM(DS:RCX).PT # PT_SS_FIRST) and (EPCM(DS:RCX).PT # PT_SS_REST)) 
THEN #PF(DS:RCX); Fl; 


(* make sure that DS:RCX points to an accessible EPC page *) 
IF ( (EPCM(DS:RCX).PENDING is not 0) or (EPCM(DS:RCS).MODIFIED is not 0) ) 
THEN 
RFLAGS.ZF := 1; 
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RAX := SGX_PAGE_NOT_DEBUGGABLE; 
GOTO DONE; 
Fl; 


(* If destination is a TCS, then make sure that the offset into the page can only point to the FLAGS field*) 
IF (( EPCM(DS:RCX). PT = PT_TCS) and ((DS:RCX) & FF8H # offset_of_FLAGS & OFF8H) ) 
THEN #GP(0); Fl; 


(* Locate the SECS for the enclave to which the DS:RCX page belongs *) 
TMP_SECS := GET_SECS_PHYS_ADDRESS(EPCM(DS:RCX),.ENCLAVESECS); 


(* make sure the enclave owning the PT_REG or PT_TCS page allow debug *) 
IF (TMP_SECS.ATTRIBUTES.DEBUG = 0) 
THEN #GP(0); Fl; 


IF ((TMP_MODE64 = 1) ) 
THEN (DS:RCX)[63:0] := RBX[63:0]; 
ELSE (DS:RCX)[31:0] := EBX[31:0]; 
Fl; 


(* clear EAX and ZF to indicate successful completion *) 
RAX := 0; 
RFLAGS.ZF := 0; 


DONE: 
(* clear flags *) 
RFLAGS.CF,PF,AF,OF,SF := 0 


Flags Affected 


ZF is set if the page is MODIFIED or PENDING; RAX contains the error code. Otherwise ZF is cleared and RAX is set 
to 0. CF, PF, AF, OF, SF are cleared. 


Protected Mode Exceptions 
#GP(0) If the address in RCS violates DS limit or access rights. 

If DS segment is unusable. 

If RCX points to a memory location not 4Byte-aligned. 

If the address in RCX points to a page belonging to a non-debug enclave. 

If the address in RCX points to a page which is not PT_TCS or PT_REG. 

If the address in RCX points to a location inside TCS that is not the FLAGS word. 
#PF(error code) If a page fault occurs in accessing memory operands. 

If the address in RCX points to a non-EPC page. 

If the address in RCX points to an invalid EPC page. 


64-Bit Mode Exceptions 
#GP(0) If RCX is non-canonical form. 
If RCX points to a memory location not 8Byte-aligned. 
If the address in RCX points to a page belonging to a non-debug enclave. 
If the address in RCX points to a page which is not PT_TCS or PT_REG. 
If the address in RCX points to a location inside TCS that is not the FLAGS word. 
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#PF(error code) If a page fault occurs in accessing memory operands. 
If the address in RCX points to a non-EPC page. 
If the address in RCX points to an invalid EPC page. 
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Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 

EAX = O6H IR VV SGX1 This leaf function measures 256 bytes of an uninitialized enclave 

ENCLS[EEXTEND] page. 
Instruction Operand Encoding 

Op/En EAX EBX RCX 
IR EEXTEND (In) | §ffective address of the SECS of the | cétactive address of a 256-byte chunk in the EPC (In) 
data chunk (In) 
Description 


This leaf function updates the MRENCLAVE measurement register of an SECS with the measurement of an EXTEND 
string compromising of “EEXTEND” || ENCLAVEOFFSET || PADDING || 256 bytes of the enclave page. This instruc- 
tion can only be executed when current privilege level is 0 and the enclave is uninitialized. 


RBX contains the effective address of the SECS of the region to be measured. The address must be the same as the 
one used to add the page into the enclave. 


RCX contains the effective address of the 256 byte region of an EPC page to be measured. The DS segment is used 
to create linear addresses. Segment override is not supported. 


EEXTEND Memory Parameter Semantics 
EPC[RCX] 


Read access by Enclave 


The instruction faults if any of the following: 


EEXTEND Faulting Conditions 
RBX does not resolve to an SECS. 


RBX points to an address not 4KBytes aligned. 


RBX does not point to an SECS page. RBX does not point to the SECS page of the data chunk. 


RCX points to an address not 256B aligned. RCX points to an unused page or a SECS. 
If SECS is locked. 


May page fault. 


RCX does not resolve in an EPC page. 
If the SECS is already initialized. 
CPL > 0. 


Concurrency Restrictions 


Table 36-23. Base Concurrency Restrictions of EEXTEND 


Base Concurrency Restrictions 
Leaf Parameter : : —- 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
EEXTEND Target [DS:RCX] Shared #GP 
SECS [DS:RBX] Concurrent 
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Table 36-24. Additional Concurrency Restrictions of EEXTEND 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EEXTEND | Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS [DS:RBX] Concurrent Exclusive #GP Concurrent 
Operation 
Temp Variables in EEXTEND Operational Flow 
Name Type Size (Bits) | Description 
TMP_SECS 64 Physical address of SECS of the enclave to which source operand belongs. 
TMP_ENCLAVEOFFS | Enclave Offset | 64 The page displacement from the enclave base address. 
ET 
TMPUPDATEFIELD SHA256 Buffer | 512 Buffer used to hold data being added to TMP_SECS.MRENCLAVE. 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1); 


IF (DS:RBX is not 4096 Byte Aligned) 


THEN #GP(0); Fl; 


IF (DS:RBX does resolve to an EPC page) 
THEN #PF(DS:RBX); FI; 


IF (DS:RCX is not 256Byte Aligned) 


THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* make sure no other Intel SGX instruction is accessing EPCM *) 
IF (Other instructions accessing EPCM) 
THEN #GP(0); Fl; 


IF (EPCM(DS:RCX). VALID = 0) 
THEN #PF(DS:RCX); Fl; 


(* make sure that DS:RCX (DST) is pointing to a PT_REG or PT_TCS or PT_SS_FIRST or PT_SS_REST *) 
IF ( (EPCM(DS:RCX).PT # PT_REG) and (EPCM(DS:RCX).PT # PT_TCS) 
and (EPCM(DS:RCX).PT # PT_SS_FIRST) and (EPCM(DS:RCX).PT # PT_SS_REST)) 
THEN #PF(DS:RCX); Fl; 


TMP_SECS := Get_SECS_ADDRESS(); 


IF (DS:RBX does not resolve to TMP_SECS) 
THEN #GP(0); Fl; 


(* make sure no other instruction is accessing MRENCLAVE or ATTRIBUTES.INIT *) 
IF ( (Other instruction accessing MRENCLAVE) or (Other instructions checking or updating the initialized state of the SECS)) 
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THEN #GP(0); Fl; 


(* Calculate enclave offset *) 
TMP_ENCLAVEOFFSET := EPCM(DS:RCX).ENCLAVEADDRESS - TMP_SECS.BASEADDR; 
TMP_ENCLAVEOFFSET := TMP_ENCLAVEOFFSET + (DS:RCX & OFFFH) 


(* Add EEXTEND message and offset to MRENCLAVE *) 

TMPUPDATEFIELD[63:0] := 00444E4554584545H; // “EEXTEND” 

TMPUPDATEFIELD[1 27:64] := TMP_ENCLAVEOFFSET; 

TMPUPDATEFIELD[51 1:1 28] := 0; // 48 bytes 

TMP_SECS.MRENCLAVE := SHA256UPDATE(TMP_SECS.MRENCLAVE, TMPUPDATEFIELD) 
INC enclave’s MRENCLAVE update counter; 


(*Add 256 bytes to MRENCLAVE, 64 byte at a time *) 

TMP_SECS.MRENCLAVE := SHA256UPDATE(TMP_SECS.MRENCLAVE, DS:RCX[51 1:0] ); 
TMP_SECS.MRENCLAVE := SHA256UPDATE(TMP_SECS.MRENCLAVE, DS:RCX[1023: 512] ); 
TMP_SECS.MRENCLAVE := SHA256UPDATE(TMP_SECS.MRENCLAVE, DS:RCX[1535: 1024] ); 
TMP_SECS.MRENCLAVE := SHA256UPDATE(TMP_SECS.MRENCLAVE, DS:RCX[2047: 1536] ); 
INC enclave's MRENCLAVE update counter by 4; 


Flags Affected 
None 


Protected Mode Exceptions 
#GP(0) If the address in RBX is outside the DS segment limit. 
If RBX points to an SECS page which is not the SECS of the data chunk. 
If the address in RCX is outside the DS segment limit. 
If RCX points to a memory location not 256Byte-aligned. 
If another instruction is accessing MRENCLAVE. 
If another instruction is checking or updating the SECS. 
If the enclave is already initialized. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If the address in RBX points to a non-EPC page. 
If the address in RCX points to a page which is not PT_TCS or PT_REG. 
If the address in RCX points to a non-EPC page. 
If the address in RCX points to an invalid EPC page. 


64-Bit Mode Exceptions 
#GP(0) If RBX is non-canonical form. 
If RBX points to an SECS page which is not the SECS of the data chunk. 
If RCX is non-canonical form. 
If RCX points to a memory location not 256 Byte-aligned. 
If another instruction is accessing MRENCLAVE. 
If another instruction is checking or updating the SECS. 
If the enclave is already initialized. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If the address in RBX points to a non-EPC page. 
If the address in RCX points to a page which is not PT_TCS or PT_REG. 
If the address in RCX points to a non-EPC page. 
If the address in RCX points to an invalid EPC page. 
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EINIT—Initialize an Enclave for Execution 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 02H IR VIV SGX1 This leaf function initializes the enclave and makes it ready to 
ENCLS[EINIT] execute enclave code. 


Instruction Operand Encoding 


Op/En EAX RBX RCX RDX 
IR EINIT (In) Error code (Out) | Address of SIGSTRUCT (In) Address of SECS (In) Address of EINITTOKEN (In) 
Description 


This leaf function is the final instruction executed in the enclave build process. After EINIT, the MRENCLAVE 
measurement is complete, and the enclave is ready to start user code execution using the EENTER instruction. 


EINIT takes the effective address of a SIGSTRUCT and EINITTOKEN. The SIGSTRUCT describes the enclave 
including MRENCLAVE, ATTRIBUTES, ISVSVN, a 3072 bit RSA key, and a signature using the included key. 
SIGSTRUCT must be populated with two values, qi and q2. These are calculated using the formulas shown below: 
qi = floor(Signature? / Modulus); 

q2 = floor((Signature? - q1 * Signature * Modulus) / Modulus); 

The EINITTOKEN contains the MRENCLAVE, MRSIGNER, and ATTRIBUTES. These values must match the corre- 


sponding values in the SECS. If the EINITTOKEN was created with a debug launch key, the enclave must be in 
debug mode as well. 


Verify 
Hashed 
ATTRIBUTES 
ATTRIBUTEMASK Gat 
MRENCLAVE 
SIGSTRUCT MRSIGNER If VALID=1,|Check 
MRSIGNER 
ATTRIBUTES — 
MRENCLAVE | _ If VALID=1, 
DS:RDX oe Check— 
EINTT <¢———_ EINITTOKEN 
Copy 
ATTRIBUTES 
DS:RCX 
Check MRENCLAVE 
ENCLAVE r 


Figure 36-1. Relationships Between SECS, SIGSTRUCT and EINITTOKEN 
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EINIT Memory Parameter Semantics 
SIGSTRUCT SECS EINITTOKEN 
Access by non-Enclave Read/Write access by Enclave Access by non-Enclave 


EINIT performs the following steps, which can be seen in Figure 36-1: 
Validates that SIGSTRUCT is signed using the enclosed public key. 
Checks that the completed computation of SECS.MRENCLAVE equals SIGSTRUCT. HASHENCLAVE. 


Checks that no reserved bits are set to 1 in SIGSTRUCT.ATTRIBUTES and no reserved bits in SIGSTRUCT.ATTRI- 
BUTESMASK are set to 0. 


Checks that no controlled ATTRIBUTES bits are set in SIGSTRUCT.ATTRIBUTES unless the SHA256 digest of 
SIGSTRUCT.MODULUS equals IA32_SGX_LEPUBKEYHASH. 


Checks that SIGSTRUCT.ATTRIBUTES equals the result of logically and-ing SIGSTRUCT.ATTRIBUTEMASK with 
SECS.ATTRIBUTES. 


If EINTTTOKEN.VALID is 0, checks that the SHA256 digest of SIGSTRUCT.MODULUS equals 
IA32_SGX_LEPUBKEYHASH. 


If EINITTOKEN.VALID is 1, checks the validity of EINITTOKEN. 
If EINITTOKEN.VALID is 1, checks that EINITTOKEN.MRENCLAVE equals SECS.MRENCLAVE. 
If EINITTOKEN.VALID is 1 and EINITTOKEN.ATTRIBUTES.DEBUG is 1, SECS.ATTRIBUTES.DEBUG must be 1. 


Commits SECS.MRENCLAVE, and sets SECS.MRSIGNER, SECS.ISVSVN, and SECS.ISVPRODID based on 
SIGSTRUCT. 


Update the SECS as Initialized. 

Periodically, EINIT polls for certain asynchronous events. If such an event is detected, it completes with failure 
code (ZF=1 and RAX = SGX_UNMASKED_EVENT), and RIP is incremented to point to the next instruction. These 
events includes external interrupts, non-maskable interrupts, system-management interrupts, machine checks, 


INIT signals, and the VMX-preemption timer. EINIT does not fail if the pending event is inhibited (e.g., external 
interrupts could be inhibited due to blocking by MOV SS blocking or by STI). 


The following bits in RFLAGS are cleared: CF, PF, AF, OF, and SF. When the instruction completes with an error, 
RFLAGS.ZF is set to 1, and the corresponding error bit is set in RAX. If no error occurs, RFLAGS.ZF is cleared and 
RAX is set to 0. 


The error codes are: 


Table 36-25. EINIT Return Value in RAX 


Error Code (see Table 36-4) Description 
No Error EINIT successful. 
SGX_INVALID_SIG_STRUCT If SIGSTRUCT contained an invalid value. 
SGX_INVALID_ATTRIBUTE If SIGSTRUCT contains an unauthorized attributes mask. 


SGX_INVALID_MEASUREMENT If SIGSTRUCT contains an incorrect measurement. 


If EINITTOKEN contains an incorrect measurement. 


SGX_INVALID_SIGNATURE If signature does not validate with enclosed public key. 

SGX_INVALID_LICENSE If license is invalid. 

SGX_INVALID_CPUSVN If license SVN is unsupported. 

SGX_UNMASKED_EVENT If an unmasked event is received before the instruction completes its operation. 
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Concurrency Restrictions 


Table 36-26. Base Concurrency Restrictions of EINIT 


ica pararieien Base klibi Restrictions _ 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EINIT SECS [DS:RCX] Shared #GP 
Table 36-27. Additional Concurrency Restrictions of ENIT 
Additional Concurrency Restrictions 
Leaf Parameter panes aoceR fies vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EINIT SECS [DS:RCX] Concurrent Exclusive #GP Concurrent 
Operation 
Temp Variables in EINIT Operational Flow 

Name Type Size Description 

TMP_SIG SIGSTRUCT 1808Bytes | Temp space for SIGSTRUCT. 

TMP_TOKEN EINITTOKEN 304Bytes | Temp space for EINITTOKEN. 

TMP_MRENCLAVE 32Bytes Temp space for calculating MRENCLAVE. 

TMP_MRSIGNER 32Bytes Temp space for calculating MRSIGNER. 

CONTROLLED_ATTRIBU | ATTRIBUTES 16Bytes Constant mask of all ATTRIBUTE bits that can only be set for authorized 
TES enclaves. 

TMP_KEYDEPENDENCIE | Buffer 224Bytes | Temp space for key derivation. 

S 

TMP_EINITTOKENKEY 16Bytes Temp space for the derived EINITTOKEN Key. 

TMP_SIG_PADDING PKCS Padding 352Bytes | The value of the top 352 bytes from the computation of Signature? 

Buffer modulo MRSIGNER. 


(* make sure SIGSTRUCT and SECS are aligned *) 
IF ( (DS:RBX is not 4KByte Aligned) or (DS:RCX is not 4KByte Aligned) ) 


THEN #GP(0); Fl; 


(* make sure the EINITTOKEN is aligned *) 
IF (DS:RDX is not 512Byte Aligned) 


THEN #GP(0); Fl; 


(* make sure the SECS is inside the EPC *) 


IF (DS:RCX does not resolve within an EPC) 


THEN #PF(DS:RCX); Fl; 


TMP_SIG[14463:0] := DS:RBX[14463:0]; // 1808 bytes 
TMP_TOKEN[2423:0] := DS:RDX[2423:0]; // 304 bytes 
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(* Verify SIGSTRUCT Header. *) 

IF ((TMP_SIG.HEADER # O6000000E1 000000000001 0O00000000h) or 
((TMP_SIG.VENDOR # 0) and (TMP_SIG.VENDOR # O00080886h) ) or 
(TMP_SIG HEADER2 # 01010000600000006000000001000000h) or 
(TMP_SIG.EXPONENT # OO000003h) or (Reserved space is not 0's) ) 
THEN 

RFLAGS.ZF := 1; 
RAX := SGX_INVALID_SIG_STRUCT; 
GOTO EXIT; 

FI; 


(* Open “Event Window” Check for Interrupts. Verify signature using embedded public key, q1, and q2. Save upper 352 bytes of the 
PKCS1.5 encoded message into the TMP_SIG_PADDING*) 
IF (interrupt was pending) THEN 
RFLAGS.ZF := 1; 
RAX := SGX_UNMASKED_EVENT; 
GOTO ExIT; 
Fl 
IF (signature failed to verify) THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_SIGNATURE; 
GOTO ExIT; 
Fl; 
(*Close “Event Window” *) 


(* make sure no other Intel SGX instruction is modifying SECS*) 
IF (Other instructions modifying SECS) 
THEN #GP(0); Fl; 


IF ( (EPCM(DS:RCX). VALID = 0) or (EPCM(DS:RCX).PT # PT_SECS) ) 
THEN #PF(DS:RCX); Fl; 


(* Verify ISVFAMILYID is not used on an enclave with KSS disabled *) 
IF ((TMP_SIG.ISVFAMILYID != 0) AND (DS:RCX.ATTRIBUTES.KSS == 0)) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_SIG_STRUCT; 
GOTO EXIT; 
FI; 


(* make sure no other instruction is accessing MRENCLAVE or ATTRIBUTES.INIT *) 
IF ( (Other instruction modifying MRENCLAVE) or (Other instructions modifying the SECS’s Initialized state)) 
THEN #GP(0); Fl; 


(* Calculate finalized version of MRENCLAVE *) 
(* SHA256 algorithm requires one last update that compresses the length of the hashed message into the output SHA256 digest *) 
TMP_ENCLAVE := SHA256FINAL( (DS:RCX).MRENCLAVE, enclave’s MRENCLAVE update count *51 2); 


(* Verify MRENCLAVE from SIGSTRUCT *) 

IF (TMP_SIG.ENCLAVEHASH # TMP_MRENCLAVE) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_MEASUREMENT; 
GOTO EXIT; 

Fl; 
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TMP_MRSIGNER := SHA256(TMP_SIG.MODULUS) 


(* if controlled ATTRIBUTES are set, SIGSTRUCT must be signed using an authorized key *) 
CONTROLLED_ATTRIBUTES := OOOO000000000020H; 
IF (((DS:RCX.ATTRIBUTES & CONTROLLED_ATTRIBUTES) # 0) and (TMP_MRSIGNER # IA32_SGXLEPUBKEYHASH) ) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO ExIT; 
Fl; 


(* Verify SIGSTRUCT.ATTRIBUTE requirements are met *) 

IF ((DS:RCX.ATTRIBUTES & TMP_SIG.AATTRIBUTEMASK) # (TMP_SIG.ATTRIBUTE & TMP_SIG.ATTRIBUTEMASK) ) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO EXIT; 

FI; 


( *Verify SIGSTRUCT.MISCSELECT requirements are met *) 
IF ((DS:RCX.MISCSELECT & TMP_SIG.MISCMASK) # (TMP_SIG.MISCSELECT & TMP_SIG.MISCMASK) ) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO EXIT 
FI; 


IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
IF (DS:RCX.CET_ATTRIBUTES & TMP_SIG.CET_ATTRIBUTES_MASK # TMP_SIG.CET_ATTRIBUTES & 
TMP_SIG.CET_ATTRIB-UTES_MASK ) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO EXIT 
Fl; 
Fl; 


(* If EINITTTOKEN.VALID[0] is 0, verify the enclave is signed by an authorized key *) 
IF (TMP_TOKEN.VALID[0] = 0) 
IF (TMP_MRSIGNER # IA32_SGXLEPUBKEYHASH) 


RFLAGS.ZF := 1; 
RAX := SGX_INVALID_EINITTOKEN; 
GOTO ExIT; 

Fl; 

GOTO COMMIT; 


Fl; 


(* Debug Launch Enclave cannot launch Production Enclaves *) 

IF ( (DS:RDX.MASKEDATTRIBUTESLE.DEBUG = 1) and (DS:RCX.ATTRIBUTES.DEBUG = 0) ) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_EINITTOKEN; 
GOTO EXIT; 

FI; 
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(* Check reserve space in EINIT token includes reserved regions and upper bits in valid field *) 
IF (TMP_TOKEN reserved space is not clear) 

RFLAGS.ZF := 1; 

RAX := SGX_INVALID_EINITTOKEN; 

GOTO EXIT; 
Fl; 


(* EINIT token must not have been created by a configuration beyond the current CPU configuration *) 
IF (TMP_TOKEN.CPUSVN must not be a configuration beyond CR_CPUSVN) 

RFLAGS.ZF := 1; 

RAX := SGX_INVALID_CPUSVN; 

GOTO EXIT; 
Fl; 


(* Derive Launch key used to calculate EINITTOKEN.MAC *) 

HARDCODED_PKCS1_5_PADDING[1 5:0] := 0100H; 

HARDCODED_PKCS1_5_PADDING[2655:16] := Sign€xtend330Byte(-1); // 330 bytes of OFFH 
HARDCODED_PKCS1_5_PADDING[2815:2656] := 2004000501020403650148866009060D30313000H; 


TMP_KEYDEPENDENCIES.KEYNAME := EINITTOKEN_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := 0; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVPRODID := TMP_TOKEN.ISVPRODIDLE; 
TMP_KEYDEPENDENCIES.ISVSVN := TMP_TOKEN.ISVSVNLE; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := CR_SGXOWNEREPOCH; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := TMP_TOKEN.MASKEDATTRIBUTESLE; 
TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := 0; 
TMP_KEYDEPENDENCIES.MRENCLAVE := 0; 
TMP_KEYDEPENDENCIES.MRSIGNER := [A32_SGXLEPUBKEYHASH; 
TMP_KEYDEPENDENCIES.KEYID := TMP_TOKEN.KEYID; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := CR_SEAL_FUSES; 
TMP_KEYDEPENDENCIES.CPUSVN := TMP_TOKEN.CPUSVNLE; 
TMP_KEYDEPENDENCIES.MISCSELECT := TMP_TOKEN.MASKEDMISCSELECTLE; 
TMP_KEYDEPENDENCIES.MISCMASK := 0; 

TMP_KEYDEPENDENCIES.PADDING := HARDCODED_PKCS1_5_ PADDING; 

TMP_KEYDEPENDENCIES.KEYPOLICY := 0; 

TMP_KEYDEPENDENCIES.CONFIGID := 0; 

TMP_KEYDEPENDENCIES.CONFIGSVN := 0; 

IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1)) 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := TMP_TOKEN.CET_MASKED_ATTRIBUTES_ LE; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES_MASK := 0; 

Fl; 


(* Calculate the derived key*) 
TMP_EINITTOKENKEY := derivekey(TMP_KEYDEPENDENCIES); 


(* Verify EINITTOKEN was generated using this CPU's Launch key and that it has not been modified since issuing by the Launch 
Enclave. Only 192 bytes of EINITTOKEN are CMACed *) 
IF (TMP_TOKEN.MAC # CMAC(TMP_EINITTOKENKEY, TMP_TOKEN[1535:0] ) ) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_EINITTOKEN; 
GOTO EXIT; 
Fl; 
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(* Verify EINITTOKEN (RDX) is for this enclave *) 

IF ((TMP_TOKEN.MRENCLAVE # TMP_MRENCLAVE) or (TMP_TOKEN.MRSIGNER # TMP_MRSIGNER) ) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_MEASUREMENT; 
GOTO EXIT; 

FI; 


(* Verify ATTRIBUTES in EINITTOKEN are the same as the enclave’s *) 
IF (TMP_TOKEN.ATTRIBUTES # DS:RCX.ATTRIBUTES) 

RFLAGS.ZF := 1; 

RAX := SGX_INVALID_EINIT_ATTRIBUTE; 

GOTO EXIT; 
FI; 


COMMIT: 

(* Commit changes to the SECS; Set ISVPRODID, ISVSVN, MRSIGNER, INIT ATTRIBUTE fields in SECS (RCX) *) 
DS:RCX.MRENCLAVE := TMP_MRENCLAVE; 

(* MRSIGNER stores a SHA256 in little endian implemented natively on x86 *) 

DS:RCX.MRSIGNER := TMP_MRSIGNER; 

DS:RCX.ISVEXTPRODID := TMP_SIG.ISVEXTPRODID; 

DS:RCX.ISVPRODID := TMP_SIG.ISVPRODID; 

DS:RCX.ISVSVN := TMP_SIG.ISVSVN; 

DS:RCX.ISVFAMILYID := TMP_SIG.ISVFAMILYID; 

DS:RCX.PADDING := TMP_SIG_PADDING; 


(* Mark the SECS as initialized *) 
Update DS:RCX to initialized; 


(* Set RAX and ZF for success*) 
RFLAGS.ZF := 0; 
RAX := 0; 

EXIT: 

RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 
ZF is cleared if successful, otherwise ZF is set and RAX contains the error code. CF, PF, AF, OF, SF are cleared. 


Protected Mode Exceptions 
#GP(0) If a memory operand is not properly aligned. 
If another instruction is modifying the SECS. 
If the enclave is already initialized. 
If the SECS.MRENCLAVE is in use. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If RCX does not resolve in an EPC page. 
If the memory address is not a valid, uninitialized SECS. 


64-Bit Mode Exceptions 

#GP(0) If a memory operand is not properly aligned. 
If another instruction is modifying the SECS. 
If the enclave is already initialized. 
If the SECS.MRENCLAVE is in use. 
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#PF(error code) If a page fault occurs in accessing memory operands. 
If RCX does not resolve in an EPC page. 
If the memory address is not a valid, uninitialized SECS. 
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ELDB/ELDU/ELDBC/ELDUC—Load an EPC Page and Mark its State 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 07H IR VIN SGX1 This leaf function loads, verifies an EPC page and marks the page 
ENCLS[ELDB] as blocked. 
EAX = 08H IR VIN SGX1 This leaf function loads, verifies an EPC page and marks the page 
ENCLS[ELDU] as unblocked. 
EAX = 12H IR VN EAX[6] This leaf function behaves lie ELDB but with improved conflict 
ENCLS[ELDBC] handling for oversubscription. 
EAX = 13H IR VIV EAX[6] This leaf function behaves like ELDU but with improved conflict 
ENCLS[ELDUC] handling for oversubscription. 
Instruction Operand Encoding 
Op/En EAX RBX RCX RDX 
ELDB/ELDU Return error Address of the PAGEINFO | Address of the EPC page Address of the version- 
IR 
(In) code (Out) (In) (In) array slot (In) 
Description 


This leaf function copies a page from regular main memory to the EPC. As part of the copying process, the page is 
cryptographically authenticated and decrypted. This instruction can only be executed when current privilege level 


is O. 


The ELDB leaf function sets the BLOCK bit in the EPCM entry for the destination page in the EPC after copying. The 
ELDU leaf function clears the BLOCK bit in the EPCM entry for the destination page in the EPC after copying. 


RBX contains the effective address of a PAGEINFO structure; RCX contains the effective address of the destination 
EPC page; RDX holds the effective address of the version array slot that holds the version of the page. 


The ELDBC/ELDUC leafs are very similar to ELDB and ELDU. They provide an error code on the concurrency conflict 
for any of the pages which need to acquire a lock. These include the destination, SECS, and VA slot. 


The table below provides additional information on the memory parameter of ELDB/ELDU leaf functions. 


ELDB/ELDU/ELDBC/ELBUC Memory Parameter Semantics 


PAGEINFO PAGEINFO.SRCPGE PAGEINFO.PCMD PAGEINFO.SECS EPCPAGE Version-Array Slot 
Non-enclave Non-enclave read Non-enclave read | Enclave read/write Read/Write access Read/Write access per- 
read access access access access permitted by Enclave mitted by Enclave 


The error codes are: 


Table 36-28. ELDB/ELDU/ELDBC/ELBUC Return Value in RAX 


Error Code (see Table 36-4) 


Description 


No Error 


ELDB/ELDU successful. 


SGX_MAC_COMPARE_FAIL 


If the MAC check fails. 
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Concurrency Restrictions 


Table 36-29. Base Concurrency Restrictions of ELDB/ELDU/ELDBC/ELBUC 


Base Concurrency Restrictions 
Leaf Parameter : - ar 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
ELDB/ELDU Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
VA [DS:RDX] Shared #GP 
SECS [DS:RBX]PAGEINFO.SECS Shared #GP 
ELDBC/ELBUC | Target [DS:RCX] Exclusive SGX_EPC_PAGE_ | EPC_PAGE_CONFLICT_ERROR 
CONFLICT 
VA [DS:RDX] Shared SGX_EPC_PAGE_ 
CONFLICT 
SECS [DS:RBX]PAGEINFO.SECS Shared SGX_EPC_PAGE_ 
CONFLICT 


Table 36-30. Additional Concurrency Restrictions of ELDB/ELDU/ELDBC/ELBUC 


Additional Concurrency Restrictions 
Leaf Parameter cicero eebee = ia ada vs. ETRACK, ETRACKC 
EMODPR, EMODT 
Access On Conflict Access On Conflict Access On Conflict 
ELDB/ELDU | Target [DS:RCX] Concurrent Concurrent Concurrent 
VA [DS:RDX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGEINFO.SECS | Concurrent Concurrent Concurrent 
ELDBC/ELBUC | Target [DS:RCX] Concurrent Concurrent Concurrent 
VA [DS:RDX] Concurrent Concurrent Concurrent 
SECS [DS:RBX]PAGEINFO.SECS | Concurrent Concurrent Concurrent 
Operation 


Temp Variables in ELDB/ELDU/ELDBC/ELBUC Operational Flow 


Name Type Size (Bits) | Description 
TMP_SRCPGE Memory page 4kKBytes 

TMP_SECS Memory page 4KBytes 

TMP_PCMD PCMD 128 Bytes 

TMP_HEADER MACHEADER 128 Bytes 

TMP_VER UINT64 64 

TMP_MAC UINT128 128 

TMP_PK UINT128 128 Page encryption/MAC key. 
SCRATCH_PCMD PCMD 128 Bytes 


(* Check PAGEINFO and EPCPAGE alignment *) 
IF ( (DS:RBX is not 32Byte Aligned) or (DS:RCX is not 4KByte Aligned) ) 
THEN #GP(0); Fl; 


Vol. 3D 36-55 


INTEL® SGX INSTRUCTION REFERENCES 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* Check VASLOT alignment *) 
IF (DS:RDX is not 8Byte aligned) 
THEN #GP(0); Fl; 


IF (DS:RDX does not resolve within an EPC) 
THEN #PF(DS:RDX); Fl; 


TMP_SRCPGE := DS:RBX.SRCPGE; 
TMP_SECS := DS:RBX.SECS; 
TMP_PCMD := DS:RBX.PCMD; 


(* Check alignment of PAGEINFO (RBX) linked parameters. Note: PCMD pointer is overlaid on top of PAGEINFO.SECINFO field *) 
IF ((DS:TMP_PCMD is not 128Byte aligned) or (DS:TMP_SRCPGE is not 4KByte aligned) ) 
THEN #GP(0); Fl; 


(* Check concurrency of EPC by other Intel SGX instructions *) 
IF (other instructions accessing EPC) 
THEN 
IF (EAX==07h) OR (EAX==08h)) (* ELDB/ELDU *) 
THEN 
IF (<<VMX non-root operation> > AND 
<<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := 
<< translation of DS:RCX produced by paging >>; 
VMCS.Guest-linear_address := DS:RCX; 
Deliver VMEXIT; 
ELSE 
#GP(0); 
Fl; 
ELSE (* ELDBC/ELDUC *) 
IF (<<VMX non-root operation> > AND 
<<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_ERROR; 
VMCS.Exit_qualification.error := SGX_EPC_PAGE_CONFLICT; 
V/MCS.Guest-physical_address := 
<< translation of DS:RCX produced by paging >>; 
VMCS.Guest-linear_address := DS:RCX; 
Deliver VMEXIT; 
ELSE 
RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO ERROR_EXIT; 
Fl; 
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Fl; 
Fl; 


(* Check concurrency of EPC and VASLOT by other Intel SGX instructions *) 
IF (Other instructions modifying VA slot) 


THEN 
IF ((EAX==07h) OR (EAX==08h)) (* ELDB/ELDU *) 
#GP(0); 
Fl; 
ELSE (* ELDBC/ELDUC *) 
RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 


RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO ERROR_EXIT; 
Fl; 


(* Verify EPCM attributes of EPC page, VA, and SECS *) 
IF (EPCM(DS:RCX).VALID = 1) 
THEN #PF(DS:RCX); Fl; 


IF ( (EPCM(DS:RDX & ~OFFFH).VALID = 0) or (EPCM(DS:RDX & ~OFFFH).PT # PT_VA) ) 
THEN #PF(DS:RDX); Fl; 


(* Copy PCMD into scratch buffer *) 
SCRATCH_PCMD[1 023: 0] := DS:TMP_PCMD[1023:0}; 


(* Zero out TMP_HEADER*) 
TMP_HEADER[sizeof(TMP_HEADER)-1: 0] := 0; 


TMP_HEADER.SECINFO := SCRATCH_PCMD.SECINFO; 
TMP_HEADER.RSVD := SCRATCH_PCMD.RSVD; 
TMP_HEADER.LINADDR := DS:RBX.LINADDR; 


(* Verify various attributes of SECS parameter *) 
IF ((TMP_HEADER.SECINFO.FLAGS.PT = PT_REG) or (TMP_HEADER.SECINFO.FLAGS.PT = PT_TCS) or 
(TMP_HEADER.SECINFO.FLAGS.PT = PT_TRIM) or 
(TMP_HEADER.SECINFO.FLAGS.PT = PT_SS_FIRST and CPUID.(EAX=12H, ECX=1):EAX[6] = 1) or 
(TMP_HEADER.SECINFO.FLAGS.PT = PT_SS_REST and CPUID.(EAX=12H, ECX=1):EAX[6] = 1)) 
THEN 
IF (DS:TMP_SECS is not 4KByte aligned) 
THEN #GP(0) Fl; 
IF (DS:TMP_SECS does not resolve within an EPC) 
THEN #PF(DS:TMP_SECS) FI; 
IF ( Other instructions modifying SECS) 
THEN 
IF ((EAX==07h) OR (EAX==08h)) (* ELDB/ELDU *) 
#GP(0); 
Fl; 
ELSE (* ELDBC/ELDUC *) 
RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO ERROR_EXIT; 
Fl; 
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Fl; 


IF ((TMP_HEADER.SECINFO.FLAGS.PT = PT_REG) or (TMP_HEADER.SECINFO.FLAGS.PT = PT_TCS) or 
(TMP_HEADER.SECINFO.FLAGS.PT = PT_TRIM) or 
(TMP_HEADER.SECINFO.FLAGS.PT = PT_SS_FIRST and CPUID.(EAX=12H, ECX=1):EAX[6] = 1) or 
(TMP_HEADER.SECINFO.FLAGS.PT = PT_SS_REST and CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1)) 
THEN 
TMP_HEADER.EID := DS: TMP_SECS.EID; 
ELSE 
(* These pages do not have any parent, and hence no EID binding *) 
TMP_HEADER.EID := 0; 
Fl; 


(* Copy 4KBytes SRCPGE to secure location *) 
DS:RCX[32767: 0] := DS:TMP_SRCPGE[32767: 0]; 
TMP_VER := DS:RDX[63:0]; 


(* Decrypt and MAC page. AES_GCM_DEC has 2 outputs, {plain text, MAC} *) 
(* Parameters for AES_GCM_DEC {Key, Counter, ..} *) 
{DS:RCX, TMP_MAC} := AES_GCM_DEC(CR_BASE_PK, TMP_VER << 32, TMP_HEADER, 128, DS:RCX, 4096); 


IF ((TMP_MAC # DS:TMP_PCMD.MAC) ) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_MAC_COMPARE_FAIL; 
GOTO ERROR_EXIT; 
Fl; 


(* Check version before committing *) 
IF (DS:RDX # 0) 
THEN #GP(0); 
ELSE 
DS:RDX := TMP_VER; 
FI; 


(* Commit EPCM changes *) 

EPCM(DS:RCX).PT := TMP_HEADER.SECINFO.FLAGS.PT; 
EPCM(DS:RCX).RWX := TMP_HEADER.SECINFO.FLAGS.RWX; 
EPCM(DS:RCX).PENDING := TMP_HEADER.SECINFO.FLAGS.PENDING; 
EPCM(DS:RCX).MODIFIED := TMP_HEADER.SECINFO.FLAGS.MODIFIED; 
EPCM(DS:RCX).PR := TMP_HEADER.SECINFO.FLAGS.PR; 
EPCM(DS:RCX).ENCLAVEADDRESS := TMP_HEADER.LINADDR; 


IF ( ((EAX = 07H) or (EAX = 12H)) and (TMP_HEADER.SECINFO.FLAGS.PT is NOT PT_SECS or PT_VA)) 
THEN 
EPCM(DS:RCX).BLOCKED := 1; 
ELSE 
EPCM(DS:RCX).BLOCKED := 0; 
Fl; 


IF (TMP_HEADER.SECINFO.FLAGS.PT is PT_SECS) 


<< store translation of DS:RCX produced by paging in SECS(DS:RCX).ENCLAVECONTEXT >> 
Fl; 
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EPCM(DS:RCX). VALID := 1; 


RAX := 0; 
RFLAGS.ZF := 0; 


ERROR_EXIT: 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 
Sets ZF if unsuccessful, otherwise cleared and RAX returns error code. Clears CF, PF, AF, OF, SF. 


Protected Mode Exceptions 
#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If the instruction’s EPC resource is in use by others. 
If the instruction fails to verify MAC. 
If the version-array slot is in use. 
If the parameters fail consistency checks. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand expected to be in EPC does not resolve to an EPC page. 
If one of the EPC memory operands has incorrect page type. 
If the destination EPC page is already valid. 


64-Bit Mode Exceptions 
#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If the instruction’s EPC resource is in use by others. 
If the instruction fails to verify MAC. 
If the version-array slot is in use. 
If the parameters fail consistency checks. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand expected to be in EPC does not resolve to an EPC page. 
If one of the EPC memory operands has incorrect page type. 
If the destination EPC page is already valid. 
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EMODPR—Restrict the Permissions of an EPC Page 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = OEH IR VIV SGX2 This leaf function restricts the access rights associated with a 
ENCLS[EMODPR] EPC page in an initialized enclave. 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EMODPR (In) Return Error Code (Out) Address of a SECINFO (In) Address of the destination EPC page (In) 
Description 


This leaf function restricts the access rights associated with an EPC page in an initialized enclave. THE RWX bits of 
the SECINFO parameter are treated as a permissions mask; supplying a value that does not restrict the page 
permissions will have no effect. This instruction can only be executed when current privilege level is 0. 


RBX contains the effective address of a SECINFO structure while RCX contains the effective address of an EPC page. 
The table below provides additional information on the memory parameter of the EMODPR leaf function. 


EMODPR Memory Parameter Semantics 


SECINFO EPCPAGE 


Read access permitted by Non Enclave Read/Write access permitted by Enclave 


The instruction faults if any of the following: 


EMODPR Faulting Conditions 
The operands are not properly aligned. If unsupported security attributes are set. 
The Enclave is not initialized. SECS is locked by another thread. 
The EPC page is locked by another thread. RCX does not contain an effective address of an EPC page in the running enclave. 
The EPC page is not valid. 


The error codes are: 


Table 36-31. EMODPR Return Value in RAX 


Error Code (see Table 36-4) Description 
No Error EMODPR successful. 
SGX_PAGE_NOT_MODIFIABLE The EPC page cannot be modified because it is in the PENDING or MODIFIED state. 
SGX_EPC_PAGE_CONFLICT Page is being written by EADD, EAUG, ECREATE, ELDU/B, EMODT, or EWB. 


Concurrency Restrictions 


Table 36-32. Base Concurrency Restrictions of EMODPR 


Base Concurrency Restrictions 


Leaf Parameter : - — 
Access On Conflict SGX_CONFLICT VM Exit Qualification 


EMODPR Target [DS:RCX] Shared #GP 
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Table 36-33. Additional Concurrency Restrictions of EMODPR 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 

Access On Conflict Access On Conflict Access On Conflict 

EMODPR _ | Target [DS:RCX] Exclusive SGX_EPC_PAGE | Concurrent Concurrent 
_CONFLICT 
Operation 

Temp Variables in EMODPR Operational Flow 
Name Type Size (bits) | Description 
TMP_SECS Effective Address 32/64 Physical address of SECS to which EPC operand belongs. 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:RBX. 


IF (DS:RBX is not 64Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


SCRATCH_SECINFO := DS:RBX; 


(* Check for misconfigured SECINFO flags*) 


IF ((SCRATCH_SECINFO reserved fields are not zero ) or 
(SCRATCH_SECINFO.FLAGS.R is 0 and SCRATCH_SECINFO.FLAGS.W is not 0) ) 


THEN #GP(0); Fl; 


(* Check concurrency with SGX1 or SGX2 instructions on the EPC page *) 
IF (SGX1 or other SGX2 instructions accessing EPC page) 


THEN #GP(0); Fl; 


IF (EPCM(DS:RCX).VALID is 0 ) 
THEN #PF(DS:RCX); Fl; 


(* Check the EPC page for concurrency *) 


IF (EPC page in use by another SGX2 instruction) 


THEN 
RFLAGS.ZF := 1; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO DONE; 
Fl; 


IF (EPCM(DS:RCX).PENDING is not 0 or (EPCM(DS:RCX).MODIFIED is not 0) ) 


THEN 
RFLAGS.ZF := 1; 
RAX := SGX_PAGE_NOT_MODIFIABLE; 
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GOTO DONE; 
Fl; 


IF (EPCM(DS:RCX).PT is not PT_REG) 
THEN #PF(DS:RCX); Fl; 


TMP_SECS := GET_SECS_ADDRESS 


IF (TMP_SECS.ATTRIBUTES.INIT = 0) 
THEN #GP(0); Fl; 


(* Set the PR bit to indicate that permission restriction is in progress *) 
EPCM(DS:RCX).PR := 1; 


(* Update EPCM permissions *) 

EPCM(DS:RCX).R := EPCM(DS:RCX).R & SCRATCH_SECINFO.FLAGS.R; 
EPCM(DS:RCX).W := EPCM(DS:RCX).W & SCRATCH_SECINFO.FLAGS.W; 
EPCM(DS:RCX).X := EPCM(DS:RCX).X & SCRATCH_SECINFO.FLAGS.X; 


RFLAGS.ZF := 0; 
RAX := 0; 


DONE: 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 


Sets ZF if page is not modifiable or if other SGX2 instructions are executing concurrently, otherwise cleared. Clears 
CF, PF, AF, OF, SF. 


Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 


64-Bit Mode Exceptions 

#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 
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EMODT—Change the Type of an EPC Page 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = OFH IR VV SGX2 This leaf function changes the type of an existing EPC page. 
ENCLS[EMODT] 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EMODT (In) Return Error Code (Out) Address of a SECINFO (In) Address of the destination EPC page (In) 
Description 


This leaf function modifies the type of an EPC page. The security attributes are configured to prevent access to the 
EPC page at its new type until a corresponding invocation of the EACCEPT leaf confirms the modification. This 
instruction can only be executed when current privilege level is 0. 


RBX contains the effective address of a SECINFO structure while RCX contains the effective address of an EPC 
page. The table below provides additional information on the memory parameter of the EMODT leaf function. 


EMODT Memory Parameter Semantics 
SECINFO EPCPAGE 


Read access permitted by Non Enclave Read/Write access permitted by Enclave 


The instruction faults if any of the following: 


EMODT Faulting Conditions 
The operands are not properly aligned. If unsupported security attributes are set. 
The Enclave is not initialized. SECS is locked by another thread. 
The EPC page is locked by another thread. RCX does not contain an effective address of an EPC page in the running enclave. 
The EPC page is not valid. 


The error codes are: 


Table 36-34. EMODT Return Value in RAX 


Error Code (see Table 36-4) Description 
No Error EMODT successful. 
SGX_PAGE_NOT_MODIFIABLE The EPC page cannot be modified because it is in the PENDING or MODIFIED state. 
SGX_EPC_PAGE_CONFLICT Page is being written by EADD, EAUG, ECREATE, ELDU/B, EMODPR, or EWB. 


Concurrency Restrictions 


Table 36-35. Base Concurrency Restrictions of EMODT 


Base Concurrency Restrictions 
Leaf Parameter —————————————— Ee 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EMODT Target [DS:RCX] Exclusive SGX_EPC_PAGE_ | EPC_PAGE_CONFLICT_ERROR 
CONFLICT 


Vol. 3D 36-63 


INTEL® SGX INSTRUCTION REFERENCES 


Table 36-36. Additional Concurrency Restrictions of EMODT 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EMODT Target [DS:RCX] Exclusive SGX_EPC_PAGE | Concurrent Concurrent 
_CONFLICT 
Operation 
Temp Variables in EMODT Operational Flow 

Name Type Size (bits) | Description 

TMP_SECS Effective Address 32/64 Physical address of SECS to which EPC operand belongs. 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:RBX. 


IF (DS:RBX is not 64Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


SCRATCH_SECINFO := DS:RBX; 


(* Check for misconfigured SECINFO flags*) 

IF ((SCRATCH_SECINFO reserved fields are not zero ) or 
\(SCRATCH_SECINFO.FLAGS.PT is PT_TCS or SCRATCH_SECINFO.FLAGS.PT is PT_TRIM) ) 
THEN #GP(0); Fl; 


(* Check concurrency with SGX1 instructions on the EPC page *) 
IF (other SGX1 instructions accessing EPC page) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO DONE; 
Fl; 


IF (EPCM(DS:RCX).VALID is 0) 
THEN #PF(DS:RCX); Fl; 


(* Check the EPC page for concurrency *) 
IF (EPC page in use by another SGX2 instruction) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO DONE; 
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Fl; 


IF ((EPCM(DS:RCX).PT is PT_REG or 
((EPCM(DS:RCX).PT is PT_TCS or PT_SS_FIRST or PT_SS_REST) and SCRATCH_SECINFO.FLAGS.PT is PT_TRIM))) 
THEN #PF(DS:RCX); Fl; 


IF (EPCM(DS:RCX).PENDING is not O or (EPCM(DS:RCX).MODIFIED is not 0) ) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_PAGE_NOT_MODIFIABLE; 
GOTO DONE; 
Fl; 


TMP_SECS := GET_SECS_ADDRESS 


IF (TMP_SECS.ATTRIBUTES.INIT = 0) 
THEN #GP(0); Fl; 


(* Update EPCM fields *) 

EPCM(DS:RCX).PR := 0; 

EPCM(DS:RCX).MODIFIED := 1; 

EPCM(DS:RCX).R := 0 

EPCM(DS:RCX).W := 0; 

EPCM(DS:RCX).X := 0 

EPCM(DS:RCX).PT := SCRATCH_SECINFO.FLAGS.PT; 


RFLAGS.ZF := 0; 
RAX := 0; 


DONE: 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 


Sets ZF if page is not modifiable or if other SGX2 instructions are executing concurrently, otherwise cleared. Clears 
CF, PF, AF, OF, SF. 


Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 


64-Bit Mode Exceptions 

#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 
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EPA—Add Version Array 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = OAH IR VIV SGX1 This leaf function adds a Version Array to the EPC. 
ENCLS[EPA] 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EPA (In) PT_VA (In, Constant) Effective address of the EPC page (In) 
Description 


This leaf function creates an empty version array in the EPC page whose logical address is given by DS:RCX, and 
sets up EPCM attributes for that page. At the time of execution of this instruction, the register RBX must be set to 
PT_VA. 


The table below provides additional information on the memory parameter of EPA leaf function. 


EPA Memory Parameter Semantics 


EPCPAGE 


Write access permitted by Enclave 


Concurrency Restrictions 


Table 36-37. Base Concurrency Restrictions of EPA 


Base Concurrency Restrictions 
Leaf Parameter . = oe 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EPA VA [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 


Table 36-38. Additional Concurrency Restrictions of EPA 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EPA VA [DS:RCX] Concurrent |L Concurrent Concurrent 
Operation 


IF (RBX # PT_VA or DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* Check concurrency with other Intel SGX instructions *) 
IF (Other Intel SGX instructions accessing the page) 
THEN 
IF (<<VMX non-root operation>> AND <<E€NABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
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THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := << translation of DS:RCX produced by paging >>; 
VMCS.Guest-linear_address := DS:RCX; 

Deliver VMEXIT; 

ELSE 
#GP(0); 

Fl; 
Fl; 


(* Check EPC page must be empty *) 
IF (EPCM(DS:RCX). VALID # 0) 
THEN #PF(DS:RCX); Fl; 


(* Clears EPC page *) 
DS:RCX[32767:0] := 0; 


PT := PT_VA; 
-ENCLAVEADDRESS := 0; 
-BLOCKED := 0; 
PENDING := O; 
.MODIFIED := 0; 

PR := 0; 

.RWX := 0; 

NALID := 1; 


EPCM(DS:RCX 
EPCM(DS:RCX 
EPCM(DS:RCX 
EPCM(DS:RCX 
EPCM(DS:RCX 
EPCM(DS:RCX 
EPCM(DS:RCX 
EPCM(DS:RCX 


OOOO OOO TS 


Flags Affected 
None 


Protected Mode Exceptions 
#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If another Intel SGX instruction is accessing the EPC page. 
If RBX is not set to PT_VA. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 
If the EPC page is valid. 


64-Bit Mode Exceptions 
#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If another Intel SGX instruction is accessing the EPC page. 
If RBX is not set to PT_VA. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 
If the EPC page is valid. 
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ERDINFO—Read Type and Status Information About an EPC Page 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 10H IR VIN EAX[6] This leaf function returns type and status information about an 
ENCLS[ERDINFO] EPC page. 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
IR ERDINEO (In) ec code | address of a RDINFO structure (In) | Address of the che EPE pede 
Description 


This instruction reads type and status information about an EPC page and returns it in a RDINFO structure. The 
STATUS field of the structure describes the status of the page and determines the validity of the remaining fields. 
The FLAGS field returns the EPCM permissions of the page; the page type; and the BLOCKED, PENDING, MODI- 
FIED, and PR status of the page. For enclave pages, the ENCLAVECONTEXT field of the structure returns the value 
of SECS.ENCLAVECONTEXT. For non-enclave pages (e.g., VA) ENCLAVECONTEXT returns 0. 


For invalid or non-EPC pages, the instruction returns an information code indicating the page's status, in addition 
to populating the STATUS field. 


ERDINFO returns an error code if the destination EPC page is being modified by a concurrent SGX instruction. 


RBX contains the effective address of a RDINFO structure while RCX contains the effective address of an EPC page. 
The table below provides additional information on the memory parameter of ERDINFO leaf function. 


ERDINFO Memory Parameter Semantics 


RDINFO 


Read/Write access permitted by Non Enclave 


EPCPAGE 


Read access permitted by Enclave 


The instruction faults if any of the following: 


ERDINFO Faulting Conditions 


A memory operand effective address is outside the DS 
segment limit (32b mode). 


A memory operand is not properly aligned. 


DS segment is unusable (32b mode). A page fault occurs in accessing memory operands. 


A memory address is in a non-canonical form (64b mode). 


The error codes are: 


Table 36-39. ERDINFO Return Value in RAX 
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Error Code Value Description 
No Error 0 ERDINFO successful. 
SGX_EPC_PAGE_CONFLICT Failure due to concurrent operation of another SGX instruction. 
SGX_PG_INVLD Target page is not a valid EPC page. 
SGX_PG_NONEPC Page is not an EPC page. 
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Concurrency Restrictions 


Table 36-40. Base Concurrency Restrictions of ERDINFO 


Base Concurrency Restrictions 
Leaf Parameter : - — 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
ERDINFO Target [DS:RCX] Shared SGX_EPC_PAGE_ 
CONFLICT 


Table 36-41. Additional Concurrency Restrictions of ERDINFO 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 

Access On Conflict Access On Conflict Access On Conflict 

ERDINFO | Target [DS:RCX] Concurrent Concurrent Concurrent 
Operation 

Temp Variables in ERDINFO Operational Flow 
Name Type Size (Bits) Description 
TMP_SECS Physical Address 64 Physical address of the SECS of the page being modified. 
TMP_RDINFO Linear Address 64 Address of the RDINFO structure. 


(* check alignment of RDINFO structure (RBX) *) 
IF (DS:RBX is not 32Byte Aligned) THEN 
#GP(0); Fl; 


(* check alignment of the EPCPAGE (RCX) *) 
IF (DS:RCX is not 4KByte Aligned) THEN 
#GP(0); Fl; 


(* check that EPCPAGE (DS:RCX) is the address of an EPC page *) 
IF (DS:RCX does not resolve within EPC) THEN 

RFLAGS.CF := 1; 

RFLAGS.ZF := 0; 

RAX := SGX_PG_NONEPC; 

goto DONE; 

FI; 


(* Check the EPC page for concurrency *) 
IF (EPC page is being modified) THEN 
RFLAGS.ZF = 1; 

RFLAGS.CF = 0; 

RAX = SGX_EPC_PAGE_CONFLICT; 
goto DONE; 

Fl; 


(* check page validity *) 


IF (EPCM(DS:RCX).VALID = 0) THEN 
RFLAGS.CF = 1; 
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RFLAGS.ZF = 0; 
RAX = SGX_PG_INVLD; 
goto DONE; 

Fl; 


(* clear the fields of the RDINFO structure *) 
TMP_RDINFO := DS:RBX; 
TMP_RDINFO.STATUS := 0; 
TMP_RDINFO.FLAGS := 0; 
TMP_RDINFO.ENCLAVECONTEXT := 0; 


(* store page info in RDINFO structure *) 
TMP_RDINFO.FLAGS.RWX := EPCM(DS:RCX).RWX; 
TMP_RDINFO.FLAGS.PENDING := EPCM(DS:RCX).PENDING; 
TMP_RDINFO.FLAGS.MODIFIED := EPCM(DS:RCX).MODIFIED; 
TMP_RDINFO.FLAGS.PR := EPCM(DS:RCX).PR; 
TMP_RDINFO.FLAGS.PAGE_TYPE := EPCM(DS:RCX).PAGE_TYPE; 
TMP_RDINFO.FLAGS.BLOCKED := EPCM(DS:RCX).BLOCKED; 


(* read SECS.ENCLAVECONTEXT for enclave child pages *) 
IF ((EPCM(DS:RCX).PAGE_TYPE = PT_REG) or 
(EPCM(DS:RCX).PAGE_TYPE = PT_TCS) or 
(EPCM(DS:RCX).PAGE_TYPE = PT_TRIM) or 
(EPCM(DS:RCX).PAGE_TYPE = PT_SS_FIRST) or 
(EPCM(DS:RCX).PAGE_TYPE = PT_SS_REST) 
) THEN 
TMP_SECS := Address of SECS for (DS:RCX); 
TMP_RDINFO.ENCLAVECONTEXT := SECS(TMP_SECS).ENCLAVECONTEXT; 
FI; 


(* populate enclave information for SECS pages *) 
IF (EPCM(DS:RCX).PAGE_TYPE = PT_SECS) THEN 
IF ((VMX non-root mode) and 
(ENABLE_EPC_VIRTUALIZATION_EXTENSIONS Execution Control = 1) 
) THEN 
TMP_RDINFO.STATUS.CHILDPRESENT := 
((SECS(DS:RCX).CHLDCNT # 0) or 
SECS(DS:RCX).VIRTCHILDCNT # 0); 
ELSE 
TMP_RDINFO.STATUS.CHILDPRESENT := (SECS(DS:RCX).CHLDCNT # 0); 
TMP_RDINFO.STATUS.VIRTCHILDPRESENT := 
(SECS(DS:RCX).VIRTCHILDCNT # 0); 
TMP_RDINFO.ENCLAVECONTEXT := SECS(DS_RCX).ENCLAVECONTEXT; 
Fl; 
Fl; 


RAX := 0; 
RFLAGS.ZF := 0; 
RFLAGS.CF := 0; 


DONE: 

(* clear flags *) 
RFLAGS.PF := 0; 
RFLAGS.AF := 0; 
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RFLAGS.OF := 0; 
RFLAGS.SF := ? 0; 


Flags Affected 

ZF is set if ERDINFO fails due to concurrent operation with another SGX instruction; otherwise cleared. 
CF is set if page is not a valid EPC page or not an EPC page; otherwise cleared. 

PF, AF, OF and SF are cleared. 


Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If DS segment is unusable. 
If a memory operand is not properly aligned. 

#PF(error code) If a page fault occurs in accessing memory operands. 


64-Bit Mode Exceptions 
#GP(0) If the memory address is in a non-canonical form. 

If a memory operand is not properly aligned. 
#PF(error code) If a page fault occurs in accessing memory operands. 
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EREMOVE—Remove a page from the EPC 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 03H IR VIN SGX1 This leaf function removes a page from the EPC. 
ENCLS[EREMOVE] 


Instruction Operand Encoding 


Op/En EAX RCX 
IR EREMOVE (In) Return error code (Out) Effective address of the EPC page (In) 
Description 


This leaf function causes an EPC page to be un-associated with its SECS and be marked as unused. This instruction 
leaf can only be executed when the current privilege level is 0. 


The content of RCX is an effective address of an EPC page. The DS segment is used to create linear address. 
Segment override is not supported. 


The instruction fails if the operand is not properly aligned or does not refer to an EPC page or the page is in use by 
another thread, or other threads are running in the enclave to which the page belongs. In addition the instruction 
fails if the operand refers to an SECS with associations. 


EREMOVE Memory Parameter Semantics 
EPCPAGE 


Write access permitted by Enclave 


The instruction faults if any of the following: 


EREMOVE Faulting Conditions 


The memory operand is not properly aligned. The memory operand does not resolve in an EPC page. 
Refers to an invalid SECS. Refers to an EPC page that is locked by another thread. 
Another Intel SGX instruction is accessing the EPC page. RCX does not contain an effective address of an EPC page. 
the EPC page refers to an SECS with associations. 


The error codes are: 


Table 36-42. EREMOVE Return Value in RAX 


Error Code (see Table 36-4) Description 
No Error EREMOVE successful. 
SGX_CHILD_PRESENT If the SECS still have enclave pages loaded into EPC. 
SGX_ENCLAVE_ACT If there are still logical processors executing inside the enclave. 
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Concurrency Restrictions 


Table 36-43. Base Concurrency Restrictions of EREMOVE 


Base Concurrency Restrictions 
Leaf Parameter : - — 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
EREMOVE Target [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 


Table 36-44. Additional Concurrency Restrictions of EREMOVE 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EREMOVE | Target [DS:RCX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EREMOVE Operational Flow 
Name Type Size (Bits) | Description 
TMP_SECS Effective Address 32/64 Effective address of the SECS destination page. 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve to an EPC page) 
THEN #PF(DS:RCX); Fl; 


TMP_SECS := Get_SECS_ADDRESS(); 


(* Check the EPC page for concurrency *) 
IF (EPC page being referenced by another Intel SGX instruction) 


THEN 
IF (<<VMX non-root operation> > AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS>>) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := << translation of DS:RCX produced by paging >>; 
VMCS.Guest-linear_address := DS:RCX; 
Deliver VMEXIT; 
ELSE 
#GP(0); 
Fl; 


Fl; 


(* if DS:RCX is already unused, nothing to do*) 

IF ( (EPCM(DS:RCX).VALID = 0) or (EPCM(DS:RCX).PT = PT_TRIM AND EPCM(DS:RCX).MODIFIED = 0)) 
THEN GOTO DONE; 

FI; 
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IF ((EPCM(DS:RCX).PT = PT_VA) OR 
((EPCM(DS:RCX).PT = PT_TRIM) AND (EPCM(DS:RCX).MODIFIED = O)) ) 
THEN 
EPCM(DS:RCX).VALID := 0; 
GOTO DONE; 
Fl; 


IF (EPCM(DS:RCX).PT = PT_SECS) 
THEN 
IF (DS:RCX has an EPC page associated with it) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_CHILD_PRESENT; 
GOTO ERROR_EXIT; 
Fl; 
(* treat SECS as having a child page when VIRTCHILDCNT is non-zero *) 
IF (<<in VMX non-root operation> > AND 
<<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS>> AND 
(SECS(DS:RCX).VIRTCHILDCNT # 0)) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_CHILD_PRESENT 
GOTO ERROR_EXIT 
Fl; 
EPCM(DS:RCX).VALID := 0; 
GOTO DONE; 
Fl; 


IF (Other threads active using SECS) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_ENCLAVE_ACT; 
GOTO ERROR_EXIT; 
FI; 


IF ( (EPCM(DS:RCX).PT is PT_REG) or (EPCM(DS:RCX).PT is PT_TCS) or (EPCM(DS:RCX).PT is PT_TRIM) or 
(EPCM(DS:RCX).PT is PT_SS_FIRST) or (EPCM(DS:RCX).PT is PT_SS_REST)) 
THEN 
EPCM(DS:RCX).VALID := 0; 
GOTO DONE; 
Fl; 


DONE: 
RAX := 0; 
RFLAGS.ZF := 0; 


ERROR_EXIT: 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 
Sets ZF if unsuccessful, otherwise cleared and RAX returns error code. Clears CF, PF, AF, OF, SF. 
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Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If another Intel SGX instruction is accessing the page. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If the memory operand is not an EPC page. 


64-Bit Mode Exceptions 
#GP(0) If the memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If another Intel SGX instruction is accessing the page. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If the memory operand is not an EPC page. 
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ETRACK—Activates EBLOCK Checks 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = OCH IR VN SGX1 This leaf function activates EBLOCK checks. 
ENCLS[ETRACK] 
Instruction Operand Encoding 
Op/En EAX RCX 
IR ETRACK (In) Return error code (Out) Pointer to the SECS of the EPC page (In) 
Description 


This leaf function provides the mechanism for hardware to track that software has completed the required TLB 
address clears successfully. The instruction can only be executed when the current privilege level is 0. 


The content of RCX is an effective address of an EPC page. 


The table below provides additional information on the memory parameter of ETRACK leaf function. 


ETRACK Memory Parameter Semantics 
EPCPAGE 


Read/Write access permitted by Enclave 


The error codes are: 


Table 36-45. ETRACK Return Value in RAX 


Error Code (see Table 36-4) 


Description 


No Error 


ETRACK successful. 


SGX_PREV_TRK_INCMPL 


All processors did not complete the previous shoot-down sequence. 


Concurrency Restrictions 


Table 36-46. Base Concurrency Restrictions of ETRACK 


Leaf Parameter 


Base Concurrency Restrictions 


Access On Conflict SGX_CONFLICT VM Exit Qualification 


ETRACK SECS [DS:RCX] 


Shared #GP 


Table 36-47. Additional Concurrency Restrictions of ETRACK 


Additional Concurrency Restrictions 


vs. EACCEPT, EACCEPTCOPY, 


Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
ETRACK SECS [DS:RCX] Concurrent Concurrent Exclusive SGX_EPC_PAGE 
_CONFLICT 
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Operation 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* Check concurrency with other Intel SGX instructions *) 
IF (Other Intel SGX instructions using tracking facility on this SECS) 


THEN 
IF (<<VMX non-root operation>> AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := TRACKING_RESOURCE_CONFLICT; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := SECS(TMP_SECS).ENCLAVECONTEXT; 
VMCS.Guest-linear_address := 0; 
Deliver VMEXIT; 
ELSE 
#GP(0); 
Fl; 


Fl; 


IF (EPCM(DS:RCX). VALID = 0) 
THEN #PF(DS:RCX); Fl; 


IF (EPCM(DS:RCX).PT # PT_SECS) 
THEN #PF(DS:RCX); Fl; 


(* All processors must have completed the previous tracking cycle”) 
IF ( (DS:RCX). TRACKING # 0) ) 
THEN 
IF (<<VMX non-root operation>> AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := TRACKING_REFERENCE_CONFLICT; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := SECS(TMP_SECS).ENCLAVECONTEXT; 
VMCS.Guest-linear_address := 0; 
Deliver VMEXIT; 
Fl; 
RFLAGS.ZF := 1; 
RAX := SGX_PREV_TRK_INCMPL; 
GOTO DONE; 
ELSE 
RAX := 0; 
RFLAGS.ZF := 0; 
Fl; 


DONE: 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 
Sets ZF if SECS is in use or invalid, otherwise cleared. Clears CF, PF, AF, OF, SF. 
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Protected Mode Exceptions 
#GP(0) If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If another thread is concurrently using the tracking facility on this SECS. 
#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 


64-Bit Mode Exceptions 

#GP(0) If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If the specified EPC resource is in use. 

#PF(error code) If a page fault occurs in accessing memory operands. 
If a memory operand is not an EPC page. 
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ETRACKC—Activates EBLOCK Checks 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = 11H IR VIV EAX[6] This leaf function activates EBLOCK checks. 
ENCLS[ETRACKC] 


Instruction Operand Encoding 


Op/En EAX RCX 
IR ares Return error code (Out) pee Ot all ERE page Address of the SECS page (In, EA) 
Description 


The ETRACKC instruction is thread safe variant of ETRACK leaf and can be executed concurrently with other CPU 
threads operating on the same SECS. 


This leaf function provides the mechanism for hardware to track that software has completed the required TLB 
address clears successfully. The instruction can only be executed when the current privilege level is 0. 


The content of RCX is an effective address of an EPC page. 
The table below provides additional information on the memory parameter of ETRACK leaf function. 


ETRACKC Memory Parameter Semantics 
EPCPAGE 


Read/Write access permitted by Enclave 


The error codes are: 


Table 36-48. ETRACKC Return Value in RAX 


Error Code Value Description 
No Error 0 ETRACKC successful. 
SGX_EPC_PAGE_CONFLICT 7 Failure due to concurrent operation of another SGX instruction. 
SGX_PG_INVLD 6 Target page is not a VALID EPC page. 
SGX_PREV_TRK_INCMPL 17 All processors did not complete the previous tracking sequence. 
SGX_TRACK_NOT_REQUIRED 27 Target page type does not require tracking. 


Concurrency Restrictions 


Table 36-49. Base Concurrency Restrictions of ETRACKC 


Base Concurrency Restrictions 
Leaf Parameter SE ———————————e—eEe——————————————————————————— 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
ETRACKC Target [DS:RCX] Shared SGX_EPC_PAGE_ 
CONFLICT 
SECS implicit Concurrent 
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Table 36-50. Additional Concurrency Restrictions of ETRACKC 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
ETRACKC | Target [DS:RCX] Concurrent Concurrent Concurrent 
SECS implicit Concurrent Concurrent Exclusive SGX_EPC_PAGE 
_CONFLICT 
Operation 
Temp Variables in ETRACKC Operational Flow 
Name Type Size (Bits) | Description 
TMP_SECS Physical Address 64 Physical address of the SECS of the page being modified. 


(* check alignment of EPCPAGE (RCX) *) 
IF (DS:RCX is not 4KByte Aligned) THEN 


#GP(0); Fl; 


(* check that EPCPAGE (DS:RCX) is the address of an EPC page *) 


IF (DS:RCX does not resolve within an EPC) THEN 


#PF(DS:RCX, PFEC.SGX); Fl; 


(* Check the EPC page for concurrency *) 
IF (EPC page is being modified) THEN 


RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 


RAX := SGX_EPC_PAGE_CONFLICT; 
goto DONE_POST_LOCK_RELEASE; 


Fl; 


(* check to make sure the page is valid *) 
IF (EPCM(DS:RCX).VALID = 0) THEN 


RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 


RAX := SGX_PG_INVLD; 


GOTO DONE; 
Fl; 


(* find out the target SECS page *) 

IF (EPCM(DS:RCX).PT is PT_REG or PT_TCS or PT_TRIM or PT_SS_FIRST or PT_SS_REST) THEN 
TMP_SECS := Obtain SECS through EPCM(DS:RCX).ENCLAVESECS; 

ELSE IF (EPCM(DS:RCX).PT is PT_SECS) THEN 
TMP_SECS := Obtain SECS through (DS:RCX); 


ELSE 
RFLAGS.ZF := 0; 
RFLAGS.CF := 1; 


RAX := SGX_TRACK_NOT_REQUIRED; 


GOTO DONE; 
Fl; 
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(* Check concurrency with other Intel SGX instructions *) 
IF (Other Intel SGX instructions using tracking facility on this SECS) THEN 
IF ((VMX non-root mode) and 
(ENABLE_EPC_VIRTUALIZATION_EXTENSIONS Execution Control = 1)) THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := TRACKING_RESOURCE_CONFLICT; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := 
SECS(TMP_SECS).ENCLAVECONTEXT; 

VMCS.Guest-linear_address := 0; 
Deliver VMEXIT; 

Fl; 


RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 
RAX := SGX_EPC_PAGE_CONFLICT; 
GOTO DONE; 
Fl; 
(* All processors must have completed the previous tracking cycle*) 
IF ((TMP_SECS). TRACKING # 0) ) 
THEN 
IF ((VMX non-root mode) and 
(ENABLE_EPC_VIRTUALIZATION_EXTENSIONS Execution Control = 1)) THEN 
VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := TRACKING_REFERENCE_CONFLICT; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := 
SECS(TMP_SECS).ENCLAVECONTEXT; 
VMCS.Guest-linear_address := 0; 
Deliver VMEXIT; 
Fl; 


RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 
RAX := SGX_PREV_TRK_INCMPL; 
GOTO DONE; 
Fl; 


RFLAGS.ZF := 0; 
RFLAGS.CF := 0; 
RAX := 0; 


DONE: 
(* clear flags *) 


RFLAGS.PF,AF,OF,SF := 0; 


Flags Affected 
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ZF is set if ETRACKC fails due to concurrent operations with another SGX instructions or target page is an invalid 


EPC page or tracking is not completed on SECS page; otherwise cleared. 
CF is set if target page is not of a type that requires tracking; otherwise cleared. 
PF, AF, OF and SF are cleared. 
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Protected Mode Exceptions 

#GP(0) If the memory operand violates access-control policies of DS segment. 
If DS segment is unusable. 
If the memory operand is not properly aligned. 

#PF(error code) If the memory operand expected to be in EPC does not resolve to an EPC page. 
If a page fault occurs in access memory operand. 


64-Bit Mode Exceptions 

#GP(0) If a memory address is in a non-canonical form. 
If a memory operand is not properly aligned. 

#PF(error code) If the memory operand expected to be in EPC does not resolve to an EPC page. 
If a page fault occurs in access memory operand. 
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EWB—Invalidate an EPC Page and Write out to Main Memory 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = OBH IR VIV SGX1 This leaf function invalidates an EPC page and writes it out to 
ENCLS[EWB] main memory. 
Instruction Operand Encoding 
Op/En EAX RBX RCX RDX 
IR EWB (In) | Error code (Out) | Address of an PAGEINFO (In) | Address of the EPC page (In) | Address of a VA slot (In) 
Description 


This leaf function copies a page from the EPC to regular main memory. As part of the copying process, the page is 
cryptographically protected. This instruction can only be executed when current privilege level is 0. 


The table below provides additional information on the memory parameter of EPA leaf function. 


EWB Memory Parameter Semantics 
PAGEINFO.SRCPGE PAGEINFO.PCMD EPCPAGE 


Non-EPC R/W access Non-EPC R/W access EPC R/W access 


VASLOT 
EPC R/W access 


PAGEINFO 
Non-EPC R/W access 


The error codes are: 


Table 36-51. EWB Return Value in RAX 
Description 


Error Code (see Table 36-4) 
No Error 
SGX_PAGE_NOT_BLOCKED 
SGX_NOT_TRACKED 
SGX_VA_SLOT_OCCUPIED 
SGX_CHILD_PRESENT 


EWB successful. 


If page is not marked as blocked. 
If EWB is racing with ETRACK instruction. 
Version array slot contained valid entry. 


Child page present while attempting to page out enclave. 


Concurrency Restrictions 


Table 36-52. Base Concurrency Restrictions of EWB 


pea pecansice Base eyo unneny Restrictions _ 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
EWB Source [DS:RCX] Exclusive #GP EPC_PAGE_CONFLICT_EXCEPTION 
VA [DS:RDX] Shared #GP 
Table 36-53. Additional Concurrency Restrictions of EWB 
Additional Concurrency Restrictions 
Leaf Parameter Ceane ‘alece pri vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EWB Source [DS:RCX] Concurrent Concurrent Concurrent 
VA [DS:RDX] Concurrent Concurrent Exclusive 


Vol. 3D 36-83 


INTEL® SGX INSTRUCTION REFERENCES 


Operation 
Temp Variables in EWB Operational Flow 
Name Type Size (Bytes) | Description 
TMP_SRCPGE Memory page 4096 
TMP_PCMD PCMD 128 
TMP_SECS SECS 4096 
TMP_BPEPOCH UINT64 8 
TMP_BPREFCOUNT UINT64 8 
TMP_HEADER MAC Header 128 
TMP_PCMD_ENCLAVEID UINT64 8 
TMP_VER UINT64 8 
TMP_PK UINT128 16 


IF ( (DS:RBX is not 32Byte Aligned) or (DS:RCX is not 4KByte Aligned) ) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


IF (DS:RDX is not 8Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RDX does not resolve within an EPC) 
THEN #PF(DS:RDX); Fl; 


(* EPCPAGE and VASLOT should not resolve to the same EPC page”) 
IF (DS:RCX and DS:RDX resolve to the same EPC page) 
THEN #GP(0); Fl; 


TMP_SRCPGE := DS:RBX.SRCPGE; 
(* Note PAGEINFO.PCMD is overlaid on top of PAGEINFO.SECINFO *) 
TMP_PCMD := DS:RBX.PCMD; 


If (DS:RBX.LINADDR # 0) OR (DS:RBX.SECS # 0) 
THEN #GP(0); Fl; 


IF ((DS:TMP_PCMD is not 128Byte Aligned) or (DS:TMP_SRCPGE is not 4KByte Aligned) ) 
THEN #GP(0); Fl; 


(* Check for concurrent Intel SGX instruction access to the page *) 
IF (Other Intel SGX instruction is accessing page) 
THEN 
IF (<<VMX non-root operation> > AND <<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS> >) 
THEN 

VMCS.Exit_reason := SGX_CONFLICT; 
VMCS.Exit_qualification.code := EPC_PAGE_CONFLICT_EXCEPTION; 
VMCS.Exit_qualification.error := 0; 
VMCS.Guest-physical_address := << translation of DS:RCX produced by paging >>; 
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VMCS.Guest-linear_address := DS:RCX; 
Deliver VMEXIT; 
ELSE 
#GP(0); 
Fl; 
Fl; 


(*Check if the VA Page is being removed or changed*) 
IF (VA Page is being modified) 
THEN #GP(0); Fl; 


(* Verify that EPCPAGE and VASLOT page are valid EPC pages and DS:RDX is VA *) 
IF (EPCM(DS:RCX).VALID = 0) 
THEN #PF(DS:RCX); Fl; 


IF ( (EPCM(DS:RDX & ~OFFFH).VALID = 0) or (EPCM(DS:RDX & ~FFFH).PT is not PT_VA) ) 
THEN #PF(DS:RDX); Fl; 


(* Perform page-type-specific exception checks *) 
IF ((EPCM(DS:RCX).PT is PT_REG) or (EPCM(DS:RCX).PT is PT_TCS) or (EPCM(DS:RCX).PT is PT_TRIM ) or 
(EPCM(DS:RCX).PT is PT_SS_FIRST ) or (EPCM(DS:RCX).PT is PT_SS_REST)) 
THEN 
TMP_SECS = Obtain SECS through EPCM(DS:RCX) 
(* Check that EBLOCK has occurred correctly *) 
IF (EBLOCK is not correct) 
THEN #GP(0); Fl; 
Fl; 


RFLAGS.ZF,CF,PF,AF,OF,SF := O; 
RAX := 0; 


(* Zero out TMP_HEADER*) 
TMP_HEADER[ sizeof(TMP_HEADER) - 1 : 0] := 0; 


(* Perform page-type-specific checks *) 
IF ( (EPCM(DS:RCX).PT is PT_REG) or (EPCM(DS:RCX).PT is PT_TCS) or (EPCM(DS:RCX).PT is PT_TRIM )or 
(EPCM(DS:RCX).PT is PT_SS_FIRST ) or (EPCM(DS:RCX).PT is PT_SS_REST)) 
THEN 
(* check to see if the page is evictable *) 
IF (EPCM(DS:RCX).BLOCKED = 0) 
THEN 
RAX := SGX_PAGE NOT_BLOCKED; 
RFLAGS.ZF := 1; 
GOTO ERROR_EXIT; 
Fl; 
(* Check if tracking done correctly *) 
IF (Tracking not correct) 
THEN 
RAX := SGX_NOT_TRACKED; 
RFLAGS.ZF := 1; 
GOTO ERROR_EXIT; 
Fl; 


(* Obtain EID to establish cryptographic binding between the paged-out page and the enclave *) 
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TMP_HEADER.EID := TMP_SECS.EID; 


(* Obtain EID as an enclave handle for software *) 
TMP_PCMD_ENCLAVEID := TMP_SECS.EID; 
ELSE IF (EPCM(DS:RCX).PT is PT_SECS) 
(*check that there are no child pages inside the enclave *) 
IF (DS:RCX has an EPC page associated with it) 
THEN 
RAX := SGX_CHILD_PRESENT; 
RFLAGS.ZF := 1; 
GOTO ERROR_EXIT; 
FI: 
(* treat SECS as having a child page when VIRTCHILDCNT is non-zero *) 
IF (<<in VMX non-root operation> > AND 
<<ENABLE_EPC_VIRTUALIZATION_EXTENSIONS>> AND 
(SECS(DS:RCX).VIRTCHILDCNT # 0)) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_CHILD_PRESENT; 
GOTO ERROR_EXIT; 
FI; 
TMP_HEADER.EID := 0; 
(* Obtain EID as an enclave handle for software *) 
TMP_PCMD_ENCLAVEID := (DS:RCX).EID; 
ELSE IF (EPCM(DS:RCX).PT is PT_VA) 
TMP_HEADER.EID := 0; // Zero is not a special value 
(* No enclave handle for VA pages*) 
TMP_PCMD_ENCLAVEID := 0; 
FI; 


TMP_HEADER.LINADDR := EPCM(DS:RCX).ENCLAVEADDRESS; 
TMP_HEADER.SECINFO.FLAGS.PT := EPCM(DS:RCX).PT; 
TMP_HEADER.SECINFO.FLAGS.RWX := EPCM(DS:RCX).RWX; 
TMP_HEADER.SECINFO.FLAGS.PENDING := EPCM(DS:RCX).PENDING; 
TMP_HEADER.SECINFO.FLAGS.MODIFIED := EPCM(DS:RCX).MODIFIED; 
TMP_HEADER.SECINFO.FLAGS.PR := EPCM(DS:RCX).PR; 


(* Encrypt the page, DS:RCX could be encrypted in place. AES-GCM produces 2 values, {ciphertext, MAC}. *) 

(* AES-GCM input parameters: key, GCM Counter, MAC_HDR, MAC_HDR_SIZE, SRC, SRC_SIZE)*) 

{DS:TMP_SRCPGE, DS:TMP_PCMD.MAC} := AES_GCM_ENC(CR_BASE_PK), (TMP_VER << 32), 
TMP_HEADER, 128, DS:RCX, 4096); 


(* Write the output *) 

Zero out DS:TMP_PCMD.SECINFO 

DS:TMP_PCMD.SECINFO.FLAGS.PT := EPCM(DS:RCX).PT; 
DS:TMP_PCMD.SECINFO.FLAGS.RWX := EPCM(DS:RCX).RWX; 
DS:TMP_PCMD.SECINFO.FLAGS.PENDING := EPCM(DS:RCX).PENDING; 
DS:TMP_PCMD.SECINFO.FLAGS.MODIFIED := EPCM(DS:RCX).MODIFIED; 
DS:TMP_PCMD.SECINFO.FLAGS.PR := EPCM(DS:RCX).PR; 
DS:TMP_PCMD.RESERVED := 0; 

DS:TMP_PCMD.ENCLAVEID := TMP_PCMD_ENCLAVEID; 
DS:RBX.LINADDR := EPCM(DS:RCX).ENCLAVEADDRESS; 


(*Check if version array slot was empty *) 
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IF ([DS.RDX]) 
THEN 
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RAX := SGX_VA_SLOT_OCCUPIED 


RFLAGS.CF := 1; 


Fl; 


(* Write version to Version Array slot *) 


[DS.RDX] := TMP_VER; 


(* Free up EPCM Entry *) 
EPCM.(DS:RCX).VALID := 0; 


ERROR_EXIT: 


Flags Affected 


ZF is set if page is not blocked, not tracked, or a child is present. Otherwise cleared. 


CF is set if VA slot is previously occupied, Otherwise cleared. 


Protected Mode Exceptions 


#GP(0) 


#PF(error code) 


If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 

If the EPC page and VASLOT resolve to the same EPC page. 

If another Intel SGX instruction is concurrently accessing either the target EPC, VA, or SECS 
pages. 

If the tracking resource is in use. 

If the EPC page or the version array page is invalid. 

If the parameters fail consistency checks. 

If a page fault occurs in accessing memory operands. 

If a memory operand is not an EPC page. 

If one of the EPC memory operands has incorrect page type. 


64-Bit Mode Exceptions 


#GP(0) 


#PF(error code) 


If a memory operand is non-canonical form. 

If a memory operand is not properly aligned. 

If the EPC page and VASLOT resolve to the same EPC page. 
If another Intel SGX instruction is concurrently accessing either the target EPC, VA, or SECS 
pages. 

If the tracking resource is in use. 

If the EPC page or the version array page in invalid. 

If the parameters fail consistency checks. 

If a page fault occurs in accessing memory operands. 

If a memory operand is not an EPC page. 

If one of the EPC memory operands has incorrect page type. 


Vol. 3D 36-87 


INTEL® SGX INSTRUCTION REFERENCES 


36.4 INTEL® SGX USER LEAF FUNCTION REFERENCE 


Leaf functions available with the ENCLU instruction mnemonic are covered in this section. In general, each instruc- 
tion leaf requires EAX to specify the leaf function index and/or additional registers specifying leaf-specific input 
parameters. An instruction operand encoding table provides details of the implicitly-encoded register usage and 


associated input/output semantics. 


In many cases, an input parameter specifies an effective address associated with a memory object inside or outside 
the EPC, the memory addressing semantics of these memory objects are also summarized in a separate table. 
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Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 05H IR VV SGX2 This leaf function accepts changes made by system software to 
ENCLU[EACCEPT] an EPC page in the running enclave. 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
IR EACCEPT (In) Return Error Code (Out) | Address of a SECINFO (In) Address of the destination EPC page (In) 
Description 


This leaf function accepts changes to a page in the running enclave by verifying that the security attributes speci- 
fied in the SECINFO match the security attributes of the page in the EPCM. This instruction leaf can only be 


executed when inside the enclave. 


RBX contains the effective address of a SECINFO structure while RCX contains the effective address of an EPC 
page. The table below provides additional information on the memory parameter of the EACCEPT leaf function. 


EACCEPT Memory Parameter Semantics 


SECINFO 


EPCPAGE (Destination) 


Read access permitted by Non Enclave 


Read access permitted by Enclave 


The instruction faults if any of the following: 


EACCEPT Faulting Conditions 


The operands are not properly aligned. 


RBX does not contain an effective address in an EPC page in the running enclave. 


The EPC page is not valid. 


The EPC page is locked by another thread. 


RCX does not contain an effective address of an EPC page in the running enclave. 
Page type is PT_REG and MODIFIED bit is 0. 


SECINFO contains an invalid request. 


Page type is PT_TCS or PT_TRIM and PENDING bit is 0 and MODIFIED bit is 1. 


If security attributes of the SECINFO page make 
the page inaccessible. 


The error codes are: 


Table 36-54. EACCEPT Return Value in RAX 


Error Code (see Table 36-4) 


Description 


No Error 


EACCEPT successful. 


SGX_PAGE_ATTRIBUTES_MISMATCH 


The attributes of the target EPC page do not match the expected values. 


SGX_NOT_TRACKED 


The OS did not complete an ETRACK on the target page. 
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Concurrency Restrictions 


Table 36-55. Base Concurrency Restrictions of EACCEPT 


Base Concurrency Restrictions 
Leaf Parameter - : ae 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EACCEPT Target [DS:RCX] Shared #GP 
SECINFO [DS:RBX] Concurrent 


Table 36-56. Additional Concurrency Restrictions of EACCEPT 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EACCEPT | Target [DS:RCX] Exclusive #GP Concurrent Concurrent 
SECINFO [DS:RBX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EACCEPT Operational Flow 
Name Type Size (bits) | Description 
TMP_SECS Effective Address 32/64 Physical address of SECS to which EPC operands belongs. 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:RBX. 


IF (DS:RBX is not 64Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RBX is not within CR_ELRANGE) 
THEN #GP(0); Fl; 


IF (DS:RBX does not resolve within an EPC) 
THEN #PF(DS:RBX); FI; 

IF ((EPCM(DS:RBX &~FFFH).VALID = 0) or (EPCM(DS:RBX &~FFFH).R = 0) or (EPCM(DS:RBX &~FFFH).PENDING # 0) or 

(EPCM(DS:RBX &~FFFH).MODIFIED # 0) or (EPCM(DS:RBX &~FFFH).BLOCKED # 0) or 

(EPCM(DS:RBX &~FFFH).PT # PT_REG) or (EPCM(DS:RBX &~FFFH).ENCLAVESECS # CR_ACTIVE_SECS) or 

(EPCM(DS:RBX &~FFFH).ENCLAVEADDRESS # (DS:RBX & FFFH)) ) 

THEN #PF(DS:RBX); FI; 


(* Copy 64 bytes of contents *) 
SCRATCH_SECINFO := DS:RBX; 


(* Check for misconfigured SECINFO flags*) 
IF (SCRATCH_SECINFO reserved fields are not zero ) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 
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IF (DS:RCX is not within CR_ELRANGE) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


(* Check that the combination of requested PT, PENDING and MODIFIED is legal *) 
IF (CPUID.(EAX=1 2H, ECX=1):E€AX[6] = 0) 

THEN 
IF (NOT (((SCRATCH_SECINFO.FLAGS.PT is PT_REG) and 
((SCRATCH_SECINFO.FLAGS.PR is 1) or 
(SCRATCH_SECINFO.FLAGS.PENDING is 1)) and 
(SCRATCH_SECINFO.FLAGS.MODIFIED is 0)) or 
((SCRATCH_SECINFO.FLAGS.PT is PT_TCS or PT_TRIM) and 
(SCRATCH_SECINFO.FLAGS.PR is 0) and 
(SCRATCH_SECINFO.FLAGS.PENDING is 0) and 
(SCRATCH_SECINFO.FLAGS.MODIFIED is 1) ))) 

THEN #GP(0); Fl 

ELSE 
IF (NOT (((SCRATCH_SECINFO.FLAGS.PT is PT_REG) AND 
((SCRATCH_SECINFO.FLAGS.PR is 1) OR 
(SCRATCH_SECINFO.FLAGS.PENDING is 1)) AND 
(SCRATCH_SECINFO.FLAGS.MODIFIED is 0)) OR 
((SCRATCH_SECINFO.FLAGS.PT is PT_TCS OR PT_TRIM) AND 
(SCRATCH_SECINFO.FLAGS.PENDING is 0) AND 
(SCRATCH_SECINFO.FLAGS.MODIFIED is 1) AND 
(SCRATCH_SECINFO.FLAGS.PR is 0)) OR 
((SCRATCH_SECINFO.FLAGS.PT is PT_SS_FIRST or PT_SS_REST) AND 
(SCRATCH_SECINFO.FLAGS.PENDING is 1) AND 
(SCRATCH_SECINFO.FLAGS.MODIFIED is 0) AND 
(SCRATCH_SECINFO.FLAGS.PR is 0)))) 

THEN #GP(0); Fl; 
Fl; 


(* Check security attributes of the destination EPC page *) 
IF ((EPCM(DS:RCX).VALID is 0) or (EPCM(DS:RCX).BLOCKED is not 0) or 
((EPCM(DS:RCX).PT is not PT_REG) and (EPCM(DS:RCX).PT is not PT_TCS) and (EPCM(DS:RCX).PT is not PT_TRIM) 
and (EPCM(DS:RCX).PT is not PT_SS_ FIRST) and (EPCM(DS:RCX).PT is not PT_SS_REST)) or 
(EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS)) 
THEN #PF((DS:RCX); FI; 


(* Check the destination EPC page for concurrency *) 
IF (EPC page in use ) 
THEN #GP(0); Fl; 


(* Re-Check security attributes of the destination EPC page *) 
IF ((EPCM(DS:RCX).VALID is 0) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) ) 
THEN #PF(DS:RCX); Fl; 


(* Verify that accept request matches current EPC page settings *) 

IF ((EPCM(DS:RCX).ENCLAVEADDRESS # DS:RCX) or (EPCM(DS:RCX).PENDING # SCRATCH_SECINFO.FLAGS.PENDING) or 
(EPCM(DS:RCX).MODIFIED # SCRATCH_SECINFO.FLAGS.MODIFIED) or (EPCM(DS:RCX).R # SCRATCH_SECINFO.FLAGS.R) or 
(EPCM(DS:RCX).W # SCRATCH_SECINFO.FLAGS.W) or (EPCM(DS:RCX).X # SCRATCH_SECINFO.FLAGS.X) or 
(EPCM(DS:RCX).PT # SCRATCH_SECINFO.FLAGS.PT) ) 
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THEN 
RFLAGS.ZF := 1; 
RAX := SGX_PAGE_ATTRIBUTES_MISMATCH; 
GOTO DONE; 
FI; 
(* Check that all required threads have left enclave *) 
IF (Tracking not correct) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_NOT_TRACKED; 
GOTO DONE; 
FI; 


(* Get pointer to the SECS to which the EPC page belongs *) 
TMP_SECS = << Obtain physical address of SECS through EPCM(DS:RCX)>> 
(* For TCS pages, perform additional checks *) 
IF (SCRATCH_SECINFO.FLAGS.PT = PT_TCS) 
THEN 
IF (DS:RCX.RESERVED # 0) #GP(0); FI; 


(* Check that TCS.FLAGS.DBGOPTIN, TCS stack, and TCS status are correctly initialized *) 
(* check that TCS.PREVSSP is 0 *) 
IF ( ((DS:RCX).FLAGS.DBGOPTIN is not 0) or ((DS:RCX).CSSA 2 (DS:RCX).NSSA) or ((DS:RCX).AEP is not 0) or ((DS:RCX).STATE is not 0) 
or ((CPUID.(EAX=12H, ECX=1):EAX[6] = 1) AND ((DS:RCX).PREVSSP != 0))) 
THEN #GP(0); Fl; 


(* Check consistency of FS & GS Limit *) 
IF ((TMP_SECS.ATTRIBUTES.MODE64BIT is 0) and ((DS:RCX.FSLIMIT & FFFH # FFFH) or (DS:RCX.GSLIMIT & FFFH # FFFH)) ) 
THEN #GP(0); Fl; 
FI; 


(* Clear PENDING/MODIFIED flags to mark accept operation complete *) 
EPCM(DS:RCX).PENDING := 0; 

EPCM(DS:RCX).MODIFIED := 0; 

EPCM(DS:RCX).PR := 0; 


(* Clear EAX and ZF to indicate successful completion *) 
RFLAGS.ZF := 0; 
RAX := 0; 


DONE: 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 
Sets ZF if page cannot be accepted, otherwise cleared. Clears CF, PF, AF, OF, SF 
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Protected Mode Exceptions 


#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 

If a memory operand is locked. 

If a page fault occurs in accessing memory operands. 

If a memory operand is not an EPC page. 

If EPC page has incorrect page type or security attributes. 


64-Bit Mode Exceptions 


#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If a memory operand is non-canonical form. 

If a memory operand is not properly aligned. 

If a memory operand is locked. 

If a page fault occurs in accessing memory operands. 

If a memory operand is not an EPC page. 

If EPC page has incorrect page type or security attributes. 
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EACCEPTCOPY—Initialize a Pending Page 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 07H IR VIV SGX2 This leaf function initializes a dynamically allocated EPC page 
ENCLU[EACCEPTCOPY] from another page in the EPC. 


Instruction Operand Encoding 


Op/En EAX RBX RCX RDX 
Return Error Code Address of the destina- Address of the 
IR EACCEPTCOPY (In) (Out) Address of a SECINFO (In) tion EPC page (In) source EPC page (In) 


Description 


This leaf function copies the contents of an existing EPC page into an uninitialized EPC page (created by EAUG). 
After initialization, the instruction may also modify the access rights associated with the destination EPC page. This 
instruction leaf can only be executed when inside the enclave. 


RBX contains the effective address of a SECINFO structure while RCX and RDX each contain the effective address 
of an EPC page. The table below provides additional information on the memory parameter of the EACCEPTCOPY 
leaf function. 


EACCEPTCOPY Memory Parameter Semantics 


SECINFO EPCPAGE (Destination) EPCPAGE (Source) 


Read access permitted by Non Enclave Read/Write access permitted by Enclave Read access permitted by Enclave 


The instruction faults if any of the following: 


EACCEPTCOPY Faulting Conditions 


The operands are not properly aligned. If security attributes of the SECINFO page make the page inaccessible. 

The EPC page is locked by another thread. If security attributes of the source EPC page make the page inaccessible. 

The EPC page is not valid. RBX does not contain an effective address in an EPC page in the running enclave. 

SECINFO contains an invalid request. RCX/RDX does not contain an effective address of an EPC page in the running 
enclave. 


The error codes are: 


Table 36-57. EACCEPTCOPY Return Value in RAX 


Error Code (see Table 36-4) Description 


No Error EACCEPTCOPY successful. 


SGX_PAGE_ATTRIBUTES_MISMATCH The attributes of the target EPC page do not match the expected values. 
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Concurrency Restrictions 


Table 36-58. Base Concurrency Restrictions of EACCEPTCOPY 


Base Concurrency Restrictions 
Leaf Parameter : - — 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
EACCEPTCOPY | Target [DS:RCX] Concurrent 
Source [DS:RDX] Concurrent 
SECINFO [DS:RBX] Concurrent 


Table 36-59. Additional Concurrency Restrictions of EACCEPTCOPY 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EACCEPTCOPY | Target [DS:RCX] Exclusive #GP Concurrent Concurrent 
Source [DS:RDX] Concurrent Concurrent Concurrent 
SECINFO [DS:RBX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EACCEPTCOPY Operational Flow 
Name Type Size (bits) | Description 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:RBX. 


IF (DS:RBX is not 64Byte Aligned) 
THEN #GP(0); Fl; 


IF ((DS:RCX is not 4KByte Aligned) or (DS:RDX is not 4KByte Aligned) ) 
THEN #GP(0); Fl; 


IF ((DS:RBX is not within CR_ELRANGE) or (DS:RCX is not within CR_ELRANGE) or (DS:RDX is not within CR_ELRANGE)) 
THEN #GP(0); Fl; 


IF (DS:RBX does not resolve within an EPC) 
THEN #PF(DS:RBX); FI; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


IF (DS:RDX does not resolve within an EPC) 
THEN #PF(DS:RDX); FI; 


IF ((EPCM(DS:RBX &~FFFH 
(EPCM(DS:RBX &~FFFH 
(EPCM(DS:RBX &~FFFH 
(EPCM(DS:RBX &~FFFH 
THEN #PF(DS:RBX); Fl; 


VALID = 0) or (EPCM(DS:RBX &~FFFH).R = 0) or (EPCM(DS:RBX &~FFFH).PENDING # 0) or 


.ENCLAVESECS # CR_ACTIVE_SECS) or 
.ENCLAVEADDRESS # DS:RBX) ) 


MODIFIED # 0) or (EPCM(DS:RBX &~FFFH).BLOCKED # 0) or (EPCM(DS:RBX &~FFFH).PT # PT_REG) or 
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(* Copy 64 bytes of contents *) 
SCRATCH_SECINFO := DS:RBX; 


(* Check for misconfigured SECINFO flags*) 

IF ((SCRATCH_SECINFO reserved fields are not zero ) or (SCRATCH_SECINFO.FLAGS.R=0) AND(SCRATCH_SECINFO.FLAGS.W#O ) or 
(SCRATCH_SECINFO.FLAGS.PT is not PT_REG) ) 
THEN #GP(0); FI; 


(* Check security attributes of the source EPC page *) 

IF ( (EPCM(DS:RDX).VALID = 0) or (EPCM(DS:RCX).R = 0) or (EPCM(DS:RDX).PENDING # 0) or (EPCM(DS:RDX).MODIFIED # 0) or 
(EPCM(DS:RDX).BLOCKED # 0) or (EPCM(DS:RDX).PT # PT_REG) or (EPCM(DS:RDX).ENCLAVESECS # CR_ACTIVE_SECS) or 
(EPCM(DS:RDX).ENCLAVEADDRESS # DS:RDX)) 

THEN #PF(DS:RDX); FI; 


(* Check security attributes of the destination EPC page *) 

IF ( (EPCM(DS:RCX).VALID = 0) or (EPCM(DS:RCX).PENDING # 1) or (EPCM(DS:RCX).MODIFIED # 0) or 
(EPCM(DS:RDX).BLOCKED # 0) or (EPCM(DS:RCX).PT # PT_REG) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) ) 
THEN 

RFLAGS.ZF := 1; 
RAX := SGX_PAGE_ATTRIBUTES_MISMATCH; 
GOTO DONE; 

Fl; 


(* Check the destination EPC page for concurrency *) 
IF (destination EPC page in use ) 
THEN #GP(0); Fl; 


(* Re-Check security attributes of the destination EPC page *) 
IF ( (EPCM(DS:RCX).VALID = 0) or (EPCM(DS:RCX).PENDING # 1) or (EPCM(DS:RCX).MODIFIED # 0) or 
(EPCM(DS:RCX).R # 1) or (EPCM(DS:RCX).W # 1) or (EPCM(DS:RCX).X # 0) or 
(EPCM(DS:RCX).PT # SCRATCH_SECINFO.FLAGS.PT) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) or 
(EPCM(DS:RCX).ENCLAVEADDRESS # DS:RCX)) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_PAGE_ATTRIBUTES_MISMATCH; 
GOTO DONE; 
Fl; 


(* Copy 4KBbytes form the source to destination EPC page*) 
DS:RCX[32767:0] := DS:RDX[32767:0]; 


(* Update EPCM permissions *) 

EPCM(DS:RCX).R := SCRATCH_SECINFO.FLAGS.R; 
EPCM(DS:RCX).W := SCRATCH_SECINFO.FLAGS.W; 
EPCM(DS:RCX).X := SCRATCH_SECINFO.FLAGS.X; 
EPCM(DS:RCX).PENDING := 0; 


RFLAGS.ZF := 0; 
RAX := 0; 


DONE: 
RFLAGS.CF,PF,AF,OF,SF := 0; 
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Sets ZF if page is not modifiable, otherwise cleared. Clears CF, PF, AF, OF, SF 


Protected Mode Exceptions 


#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 

If a memory operand is locked. 

If a page fault occurs in accessing memory operands. 

If a memory operand is not an EPC page. 

If EPC page has incorrect page type or security attributes. 


64-Bit Mode Exceptions 


#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If a memory operand is non-canonical form. 

If a memory operand is not properly aligned. 

If a memory operand is locked. 

If a page fault occurs in accessing memory operands. 

If a memory operand is not an EPC page. 

If EPC page has incorrect page type or security attributes. 
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EENTER—Enters an Enclave 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 02H IR VV SGX1 This leaf function is used to enter an enclave. 
ENCLU[EENTER] 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
Content of RBX.CSSA Address of IP following 
IR EENTER (In) (Out) Address of a TCS (In) Address of AEP (In) EENTER (Out) 
Description 


The ENCLU[EENTER] instruction transfers execution to an enclave. At the end of the instruction, the logical 
processor is executing in enclave mode at the RIP computed as EnclaveBase + TCS.OENTRY. If the target address 
is not within the CS segment (32-bit) or is not canonical (64-bit), a #GP(0O) results. 


EENTER Memory Parameter Semantics 


TCS 


Enclave access 


EENTER is a serializing instruction. The instruction faults if any of the following occurs: 


Address in RBX is not properly aligned. 


locked. 


TCS pointed to by RBX is not valid or available or 


Any TCS.FLAGS's must-be-zero bit is not zero. 


Current 32/64 mode does not match the enclave mode in 
SECS.ATTRIBUTES.MODE64. 


The SECS is in use. 


Any one of DS, ES, CS, SS is not zero. 


Either of TCS-specified FS and GS segment is not a subsets of the current DS 
segment. 


If XSAVE available, CR4.0SXSAVE = 0, but SECS.ATTRIBUTES.XFRM # 3. 


CR4.OSFXSR # 1. 


If CR4.0SXSAVE = 1, SECS.ATTRIBUTES.XFRM is not a subset of XCRO. 


The following operations are performed by EENTER: 
¢ RSP and RBP are saved in the current SSA frame on EENTER and are automatically restored on EEXIT or 


interrupt. 


¢ The AEP contained in RCX is stored into the TCS for use by AEXs.FS and GS (including hidden portions) are 
saved and new values are constructed using TCS.OFSBASE/GSBASE (32 and 64-bit mode) and 
TCS.OFSLIMIT/GSLIMIT (32-bit mode only). The resulting segments must be a subset of the DS segment. 


° If CR4.O0SXSAVE == 1, XCRO is saved and replaced by SECS.ATTRIBUTES.XFRM. The effect of RFLAGS.TF 
depends on whether the enclave entry is opt-in or opt-out (see Section 38.1.2): 


— On opt-out entry, TF is saved and cleared (it is restored on EEXIT or AEX). Any attempt to set TF via a POPF 
instruction while inside the enclave clears TF (see Section 38.2.5). 


— On opt-in entry, a single-step debug exception is pended on the instruction boundary immediately after 


EENTER (see Section 38.2. 


2), 


¢ All code breakpoints that do not overlap with ELRANGE are also suppressed. If the entry is an opt-out entry, all 
code and data breakpoints that overlap with the ELRANGE are suppressed. 


* On opt-out entry, a number of performance monitoring counters and behaviors are modified or suppressed (see 


Section 38.2.3): 
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— All performance monitoring activity on the current thread is suppressed except for incrementing and firing 
of FIXED_CTR1 and FIXED_CTR2. 


— PEBS is suppressed. 


— AnyThread counting on other threads is demoted to MyThread mode and IA32_PERF_GLOBAL_STATUS[60] 
on that thread is set 


— If the opt-out entry on a hardware thread results in suppression of any performance monitoring, then the 
processor sets IA32_PERF_GLOBAL_STATUS[60] and IA32_PERF_GLOBAL_STATUS[63]. 


Concurrency Restrictions 


Table 36-60. Base Concurrency Restrictions of EENTER 

Base Concurrency Restrictions 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EENTER TCS [DS:RBX] Shared #GP 


Leaf Parameter 


Table 36-61. Additional Concurrency Restrictions of EENTER 


Additional Concurrency Restrictions 
Leaf Parameter aioe pirenes phoake vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EENTER TCS [DS:RBX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in EENTER Operational Flow 
Name Type Size (Bits) | Description 
TMP_FSBASE Effective Address 32/64 Proposed base address for FS segment. 
TMP_GSBASE Effective Address 32/64 Proposed base address for FS segment. 
TMP_FSLIMIT Effective Address 32/64 Highest legal address in proposed FS segment. 
TMP_GSLIMIT Effective Address 32/64 Highest legal address in proposed GS segment. 
TMP_XSIZE integer 64 Size of XSAVE area based on SECS.ATTRIBUTES.XFRM. 
TMP_SSA_PAGE Effective Address 32/64 Pointer used to iterate over the SSA pages in the current frame. 
TMP_GPR Effective Address 32/64 Address of the GPR area within the current SSA frame. 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1)); 


(* Make sure DS is usable, expand up *) 
IF (TMP_MODE64 = 0 and (DS not usable or ( ( DS[S] = 1) and (DS[bit 11] = 0) and DS[bit 10] = 1) )) 
THEN #GP(0); Fl; 


(* Check that CS, SS, DS, ES.base is 0 *) 
IF (TMP_MODE64 = 0) 
THEN 
IF 
IF 
IF 
IF 


CS.base # O or DS.base # 0) #GP(0); Fl; 
ES usable and ES.base # 0) #GP(0); FI; 
SS usable and SS.base # 0) #GP(0); FI; 
SS usable and SS.B = 0) #GP(0); FI; 


ee eS 
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Fl; 


IF (DS:RBX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RBX does not resolve within an EPC) 
THEN #PF(DS:RBX); FI; 


(* Check AEP is canonical*) 
IF (TMP_MODE64 = 1 and (CS:RCX is not canonical) ) 
THEN #GP(0); Fl; 


(* Check concurrency of TCS operation*) 
IF (Other Intel SGX instructions is operating on TCS) 
THEN #GP(0); Fl; 


(* TCS verification *) 
IF (EPCM(DS:RBX).VALID = 0) 
THEN #PF(DS:RBX); FI; 


IF (EPCM(DS:RBX).BLOCKED = 1) 
THEN #PF(DS:RBX); Fl; 


IF ( (EPCM(DS:RBX).ENCLAVEADDRESS # DS:RBX) or (EPCM(DS:RBX).PT # PT_TCS) ) 
THEN #PF(DS:RBX); Fl; 


IF ((EPCM(DS:RBX).PENDING = 1) or (EPCM(DS:RBX).MODIFIED = 1)) 
THEN #PF(DS:RBX); Fl; 


IF ( (DS:RBX).0SSA is not 4KByte Aligned) 
THEN #GP(0); Fl; 


(* Check proposed FS and GS *) 
IF ( ( (DS:RBX).OFSBASE is not 4KByte Aligned) or ( (DS:RBX).OGSBASE is not 4KByte Aligned) ) 
THEN #GP(0); Fl; 


(* Get the SECS for the enclave in which the TCS resides *) 
TMP_SECS := Address of SECS for TCS; 


(* Check proposed FS/GS segments fall within DS *) 
IF (TMP_MODE64 = 0) 
THEN 
TMP_FSBASE := (DS:RBX).OFSBASE + TMP_SECS.BASEADDR; 
TMP_FSLIMIT := (DS:RBX).OFSBASE + TMP_SECS.BASEADDR + (DS:RBX).FSLIMIT; 
TMP_GSBASE := (DS:RBX).OGSBASE + TMP_SECS.BASEADDR; 
TMP_GSLIMIT := (DS:RBX).OGSBASE + TMP_SECS.BASEADDR + (DS:RBX).GSLIMIT; 
(* if FS wrap-around, make sure DS has no holes*) 
IF (TMP_FSLIMIT < TMP_FSBASE) 
THEN 
IF (DS.limit < 4GB) THEN #GP(0); FI; 
ELSE 
IF (TMP_FSLIMIT > DS.limit) THEN #GP(0); FI; 
FI; 
(* if GS wrap-around, make sure DS has no holes*) 
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IF (TMP_GSLIMIT < TMP_GSBASE) 
THEN 
IF (DS.limit < 4GB) THEN #GP(0); FI; 
ELSE 
IF (TMP_GSLIMIT > DS.limit) THEN #GP(0); FI; 
Fl; 
ELSE 
TMP_FSBASE := (DS:RBX).OFSBASE + TMP_SECS.BASEADDR; 
TMP_GSBASE := (DS:RBX).OGSBASE + TMP_SECS.BASEADDR; 
IF ((TMP_FSBASE is not canonical) or (TMP_GSBASE is not canonical)) 
THEN #GP(0); Fl; 
FI; 


(* Ensure that the FLAGS field in the TCS does not have any reserved bits set *) 
IF ( ( (DS:RBX).FLAGS & FFFFFFFFFFFFFFFEH) # 0) 
THEN #GP(0); Fl; 


(* SECS must exist and enclave must have previously been EINITted *) 
IF (the enclave is not already initialized) 
THEN #GP(0); Fl; 


(* make sure the logical processor's operating mode matches the enclave *) 
IF ((TMP_MODE64 # TMP_SECS.ATTRIBUTES.MODE64BIT) ) 
THEN #GP(0); Fl; 


IF (CR4.0SFXSR = 0) 
THEN #GP(0); Fl; 


(* Check for legal values of SECS.ATTRIBUTES.XFRM *) 
IF (CR4.0SXSAVE = 0) 
THEN 
IF (TMP_SECS.ATTRIBUTES.XFRM # 03H) THEN #GP(0); Fl; 
ELSE 
IF ((TMP_SECS.ATTRIBUTES.XFRM & XCRO) # TMP_SECS.ATTRIBUES.XFRM) THEN #GP(0); FI; 
FI; 


(* Make sure the SSA contains at least one more frame *) 
IF ( (DS:RBX).CSSA 2 (DS:RBX).NSSA) 
THEN #GP(0); Fl; 


(* Compute linear address of SSA frame *) 
TMP_SSA := (DS:RBX).OSSA + TMP_SECS.BASEADDR + 4096 * TMP_SECS.SSAFRAMESIZE * (DS:RBX).CSSA; 
TMP_XSIZE := compute_XSAVE_frame_size(TMP_SECS.ATTRIBUTES.XFRM); 


FOR EACH TMP_SSA_PAGE = TMP_SSA to TMP_SSA + TMP_XSIZE 
(* Check page is read/write accessible *) 
Check that DS:TMP_SSA_PAGE is read/write accessible; 
If a fault occurs, release locks, abort and deliver that fault; 


IF (DS:TMP_SSA_PAGE does not resolve to EPC page) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 

IF (EPCM(DS:TMP_SSA_PAGE).VALID = 0) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 

IF (EPCM(DS:TMP_SSA_PAGE).BLOCKED = 1) 
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THEN #PF(DS:TMP_SSA_PAGE); Fl; 
IF (EPCM(DS:TMP_SSA_PAGE).PENDING = 1) or (EPCM(DS:TMP_SSA_PAGE).MODIFIED = 1)) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 
IF (( EPCM(DS:TMP_SSA_PAGE).ENCLAVEADDRESS # DS:TMP_SSA_PAGE) or (EPCM(DS:TMP_SSA_PAGE).PT # PT_REG) or 
(EPCM(DS:TMP_SSA_PAGE).ENCLAVESECS # EPCM(DS:RBX).ENCLAVESECS) or 
(EPCM(DS:TMP_SSA_PAGE).R = 0) or (EPCM(DS:TMP_SSA_PAGE).W = 0) ) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 
CR_XSAVE_PAGE_n := Physical_Address(DS:TMP_SSA_PAGE); 
ENDFOR 


(* Compute address of GPR area*) 
TMP_GPR := TMP_SSA + 4096 * DS:TMP_SECS.SSAFRAMESIZE - sizeof(GPRSGX_AREA); 
If a fault occurs; release locks, abort and deliver that fault; 


IF (DS:TMP_GPR does not resolve to EPC page) 
THEN #PF(DS:TMP_GPR); Fl; 

IF (EPCM(DS:TMP_GPR).VALID = 0) 

THEN #PF(DS:TMP_GPR); Fl; 
IF (EPCM(DS:TMP_GPR).BLOCKED = 1) 
THEN #PF(DS:TMP_GPR); Fl; 

IF ((EPCM(DS:TMP_GPR).PENDING = 1) or (EPCM(DS:TMP_GPR).MODIFIED = 1)) 
THEN #PF(DS:TMP_GPR); Fl; 

IF (( EPCM(DS:TMP_GPR).ENCLAVEADDRESS # DS:TMP_GPR) or (EPCM(DS:TMP_GPR).PT # PT_REG) or 
(EPCM(DS:TMP_GPR).ENCLAVESECS EPCM(DS:RBX).ENCLAVESECS) or 
(EPCM(DS:TMP_GPR).R = 0) or (EPCM(DS:TMP_GPR).W = 0) ) 

THEN #PF(DS:TMP_GPR); Fl; 


IF (TMP_MODE64 = 0) 
THEN 
IF (TMP_GPR + (GPR_SIZE -1) is not in DS segment) THEN #GP(0); FI; 
Fl; 


CR_GPR_PA := Physical_Address (DS: TMP_GPR); 


(* Validate TCS.OENTRY *) 
TMP_TARGET := (DS:RBX).OENTRY + TMP_SECS.BASEADDR; 
IF (TMP_MODE64 = 1) 
THEN 
IF (TMP_TARGET is not canonical) THEN #GP(0); FI; 
ELSE 
IF (TMP_TARGET > CS limit) THEN #GP(0); Fl; 
FI; 


(* Ensure the enclave is not already active and this thread is the only one using the TCS*) 
IF (DS:RBX.STATE = ACTIVE) 
THEN #GP(0); Fl; 


TMP_IA32_U_CET := 0 
TMP_SSP : = 0 


IF CPUID.(EAX=12H, ECX=1):EAX[6] = 1 
THEN 
IF (CR4.CET = 0) 
THEN 
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(* If part does not support CET or CET has not been enabled and enclave requires CET then fail *) 
IF (TMP_SECS.CET_ATTRIBUTES # 0 OR TMP_SECS.CET_LEG_BITMAP_OFFSET # 0 ) #GP(O); FI; 
Fl; 
(* If indirect branch tracking or shadow stacks enabled but CET state save area is not 16B aligned then fail EENTER *) 
IF (TMP_SECS.CET_ATTRIBUTES.SH_STK_EN = 1 OR TMP_SECS.CET_ATTRIBUTES.ENDBR_EN = 1 ) 
THEN 
IF (DS:RBX.OCETSSA is not 16B aligned) #GP(0); FI; 
Fl; 


IF (TMP_SECS.CET_ATTRIBUTES.SH_STK_EN OR TMP_SECS.CET_ATTRIBUTES.ENDBR_EN) 
THEN 

(* Setup CET state from SECS, note tracker goes to IDLE *) 

TMP_IA32_U_CET = TMP_SECS.CET_ATTRIBUTES; 

IF (TMP_IA32_U_CET.LEG_IwW_EN = 1 AND TMP_IA32_U_CET.ENDBR_EN = 1 ) 

THEN 

TMP_IA32_U_CET := TMP_IA32_U_CET + TMP_SECS.BASEADDR; 
TMP_IA32_U_CET := TMP_IA32_U_CET + TMP_SECS.CET_LEG_BITMAP_BASE; 


Fl; 


(* Compute linear address of what will become new CET state save area and cache its PA *) 
TMP_CET_SAVE_AREA = DS:RBX.OCETSSA + TMP_SECS.BASEADDR + (DS:RBX.CSSA) * 16 
TMP_CET_SAVE_PAGE = TMP_CET_SAVE_AREA & ~OxFFF; 


Check the TMP_CET_SAVE_PAGE page is read/write accessible 
If fault occurs release locks, abort and deliver fault 


(* Read the EPCM VALID, PENDING, MODIFIED, BLOCKED and PT fields atomically *) 
IF ((DS:TMP_CET_SAVE_PAGE Does NOT RESOLVE TO EPC PAGE) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).VALID = 0) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).PENDING = 1) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).MODIFIED = 1) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).BLOCKED = 1) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).R = 0) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).W = 0) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).ENCLAVEADDRESS # DS:TMP_CET_SAVE_PAGE) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).PT # PT_SS_REST) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).ENCLAVESECS # EPCM(DS:RBX).ENCLAVESECS)) 

THEN 

#PF(DS:TMP_CET_SAVE_PAGE); 


YH YS aS YS aS aS DS a 


Fl; 


CR_CET_SAVE_AREA_PA := Physical address(DS:TMP_CET_SAVE_AREA) 


IF TMP_IA32_U_CET.SH_STK_EN = 1 
THEN 
TMP_SSP = TCS.PREVSSP; 


Fl; 
Fl; 
Fl; 


CR_ENCLAVE_MODE := 1; 


CR_ACTIVE_SECS := TMP_SECS; 
CR_ELRANGE := (TMPSECS.BASEADDR, TMP_SECS.SIZE); 
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(* Save state for possible AEXs *) 
CR_TCS_PA := Physical_Address (DS:RBX); 
CR_TCS_LA := RBX; 

CR_TCS_LA.AEP := RCX; 


(* Save the hidden portions of FS and GS *) 
CR_SAVE_FS_selector := FS.selector; 
CR_SAVE_FS_base := FS.base; 
CR_SAVE_FS_limit := FS.limit; 
CR_SAVE_FS_access_rights := FS.access_rights; 
CR_SAVE_GS_selector := GS.selector; 
CR_SAVE_GS_base := GS.base; 
CR_SAVE_GS_limit := GS limit; 
CR_SAVE_GS_access_rights := GS.access_rights; 


(* If XSAVE is enabled, save XCRO and replace it with SECS.ATTRIBUTES.XFRM*) 
IF (CR4.0SXSAVE = 1) 

CR_SAVE_XCRO := XCRO; 

XCRO := TMP_SECS.ATTRIBUTES.XFRM; 
FI; 


RCX := RIP; 

RIP := TMP_TARGET; 

RAX := (DS:RBX).CSSA; 

(* Save the outside RSP and RBP so they can be restored on interrupt or EEXIT *) 
DS:TMP_SSA.U_RSP := RSP; 

DS:TMP_SSA.U_RBP := RBP; 


(* Do the FS/GS swap *) 
FS.base := TMP_FSBASE; 
FS. limit := DS:RBX.FSLIMIT; 
FS.type := 00016; 

FS.W := DS.W; 

FS.S := 1; 

FS.DPL := DS.DPL; 
FS.G:= 1; 

FS.B := 1; 

FS.P := 1; 

FS.AVL := DS.AVL; 
FS.L:= DS.L; 

FS.unusable := 0; 
FS.selector := OBH; 


GS.base := TMP_GSBASE; 
GS.limit := DS:RBX.GSLIMIT; 
GS.type := 0001b; 

GS.W := DS.W; 

GSS := 1; 

GS.DPL := DS.DPL; 

GS.G := 1; 

GS.B := 1; 

GS.P := 1; 

GS.AVL := DS.AVL; 

GS.L := DS.L; 
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GS.unusable := 0; 
GS.selector := OBH; 


CR_DBGOPTIN := TCS.FLAGS.DBGOPTIN; 
Suppress_all_code_breakpoints_that_are_outside_ELRANGE; 


IF (CR_DBGOPTIN = 0) 

THEN 
Suppress_all_code_breakpoints_that_overlap_with_ELRANGE; 
CR_SAVE_TF := RFLAGS.TF; 

RFLAGS.TF := 0; 

Suppress_monitor_trap_flag for the source of the execution of the enclave; 
Suppress any pending debug exceptions; 

Suppress any pending MTF VM exit; 

ELSE 

IF RFLAGS.TF = 1 

THEN pend a single-step #DB at the end of EENTER; FI; 
IF the “monitor trap flag” VM-execution control is set 

THEN pend an MTF VM exit at the end of EENTER; Fl; 


Fl; 


IF ((CPUID.(EAX=7H, ECX=0):EDX[CET_IBT] = 1) OR (CPUID.(EAX=7H, ECX=0):ECX[CET_SS] = 1) 
THEN 
(* Save enclosing application CET state into save registers *) 
CR_SAVE_IA32_U_CET := IA32_U_CET 
(* Setup enclave CET state *) 
IF CPUID.(EAX=07H, ECX=00h):ECX[CET_SS] = 1 
THEN 
CR_SAVE_SSP := SSP 
Fl; 


IA32_U_CET := TMP_IA32_U_CET; 


Fl; 


Flush_linear_context; 
Allow_front_end_to_begin_fetch_at_new_RIP; 


Flags Affected 
RFLAGS.TF is cleared on opt-out entry 


Protected Mode Exceptions 
#GP(0) If DS:RBX is not page aligned. 
If the enclave is not initialized. 


If part or all of the FS or GS segment specified by TCS is outside the DS segment or not prop- 
erly aligned. 


If the thread is not in the INACTIVE state. 

If CS, DS, ES or SS bases are not all zero. 

If executed in enclave mode. 

If any reserved field in the TCS FLAG is set. 

If the target address is not within the CS segment. 
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#PF(error code) 


If CR4.OSFXSR = 0. 

If CR4.0SXSAVE = 0 and SECS.ATTRIBUTES.XFRM # 3. 

If CR4.0SXSAVE = land SECS.ATTRIBUTES.XFRM is not a subset of XCRO. 
If a page fault occurs in accessing memory. 

If DS:RBX does not point to a valid TCS. 


If one or more pages of the current SSA frame are not readable/writable, or do not resolve to 
a valid PT_REG EPC page. 


64-Bit Mode Exceptions 


#GP(0) 


#PF(error code) 
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If DS:RBX is not page aligned. 

If the enclave is not initialized. 

If the thread is not in the INACTIVE state. 
If CS, DS, ES or SS bases are not all zero. 
If executed in enclave mode. 


If part or all of the FS or GS segment specified by TCS is outside the DS segment or not prop- 
erly aligned. 


If the target address is not canonical. 

If CR4.OSFXSR = 0. 

If CR4.0SXSAVE = 0 and SECS.ATTRIBUTES.XFRM # 3. 

If CR4.0SXSAVE = land SECS.ATTRIBUTES.XFRM is not a subset of XCRO. 
If a page fault occurs in accessing memory operands. 

If DS:RBX does not point to a valid TCS. 


If one or more pages of the current SSA frame are not readable/writable, or do not resolve to 
a valid PT_REG EPC page. 
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EEXIT—Exits an Enclave 


Opcode/ Op/En 64/32 CPUID Description 
Instruction bitMode Feature 
Support Flag 
EAX = 04H IR VIV SGX1 This leaf function is used to exit an enclave. 
ENCLU[EEXIT] 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EEXIT (In) Target address outside the enclave (In) Address of the current AEP (Out) 
Description 


The ENCLU[EEXIT] instruction exits the currently executing enclave and branches to the location specified in RBX. 
RCX receives the current AEP. If RBX is not within the CS (32-bit mode) or is not canonical (64-bit mode) a #GP(0) 
results. 


EEXIT Memory Parameter Semantics 
Target Address 


Non-Enclave read and execute access 


If RBX specifies an address that is inside the enclave, the instruction will complete normally. The fetch of the next 
instruction will occur in non-enclave mode, but will attempt to fetch from inside the enclave. This fetch returns a 
fixed data pattern. 


If secrets are contained in any registers, it is responsibility of enclave software to clear those registers. 


If XCRO was modified on enclave entry, it is restored to the value it had at the time of the most recent EENTER or 
ERESUME. 


If the enclave is opt-out, RFLAGS.TF is loaded from the value previously saved on EENTER. 
Code and data breakpoints are unsuppressed. 
Performance monitoring counters are unsuppressed. 


Concurrency Restrictions 


Table 36-62. Base Concurrency Restrictions of EEXIT 

Base Concurrency Restrictions 
Access On Conflict SGX_CONEFLICT VM Exit Qualification 
EEXIT Concurrent 


Leaf Parameter 


Table 36-63. Additional Concurrency Restrictions of EEXIT 
Additional Concurrency Restrictions 


vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EEXIT Concurrent Concurrent Concurrent 
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Operation 
Temp Variables in EEXIT Operational Flow 
Name Type Size (Bits) | Description 
TMP_RIP Effective Address 32/64 Saved copy of CRIP for use when creating LBR. 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1); 


IF (TMP_MODE64 = 1) 
THEN 
IF (RBX is not canonical) THEN #GP(0); FI; 
ELSE 
IF (RBX > CS limit) THEN #GP(0); FI; 
Fl; 


TMP_RIP := CRIP; 
RIP := RBX; 


(* Return current AEP in RCX *) 
RCX := CR_TCS_PA.AEP; 


(* Do the FS/GS swap *) 

FS.selector := CR_SAVE_FS.selector; 

FS.base := CR_SAVE_FS.base; 

FS.limit := CR_SAVE_FS limit; 

FS.access_rights := CR_SAVE_FS.access_rights; 
GS.selector := CR_SAVE_GS.selector; 

GS.base := CR_SAVE_GS.base; 

GS.limit = CR_SAVE_GS. limit; 

GS.access_rights := CR_SAVE_GS.access_rights; 


(* Restore XCRO if needed *) 
IF (CR4.0SXSAVE = 1) 

XCRO := CR_SAVE__XCRO; 
FI; 


Unsuppress_all_code_breakpoints_that_are_outside_ELRANGE; 


IF (CR_DBGOPTIN = 0) 
THEN 

UnSuppress_all_code_breakpoints_that_overlap_with_ELRANGE; 
Restore suppressed breakpoint matches; 
RFLAGS.TF := CR_SAVE_TF; 
UnSuppress_montior_trap_flag; 
UnSuppress_LBR_Generation; 
UnSuppress_performance monitoring_activity; 
Restore performance monitoring counter AnyThread demotion to MyThread in enclave back to AnyThread 


Fl; 


IF RFLAGS.TF = 1 
THEN Pend Single-Step #DB at the end of EEXIT; 
Fl; 
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IF the “monitor trap flag” VM-execution control is set 
THEN pend a MTF VM exit at the end of EEXIT; 
Fl; 


IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
THEN 
(* Record PREVSSP *) 
IF (IA32_U_CET.SH_STK_EN == 1) 
THEN CR_TCS_PA.PREVSSP = SSP; Fl; 
Fl; 


IF ((CPUID.(EAX=7H, ECX=0):EDX[CET_IBT] = 1) OR (CPUID.(EAX=7, ECX=0):ECX[CET_SS] = 1) 
THEN 
(* Restore enclosing app's CET state from the save registers *) 
IA32_U_CET := CR_SAVE_IA32_U_CET; 
IF CPUID.(EAX=07H, ECX=00h):ECX[CET_SS] = 1 
THEN SSP := CR_SAVE_SSP; FI; 


(* Update enclosing app's TRACKER if enclosing app has indirect branch tracking enabled *) 
IF (CR4.CET = 1 AND IA32_U_CET.ENDBR_EN = 1) 
THEN 
IA32_U_CET.TRACKER := WAIT_FOR_ENDBRANCH; 
IA32_U_CET.SUPPRESS := 0 
Fl; 
Fl; 


CR_ENCLAVE_MODE := 0; 
CR_TCS_PA.STATE := INACTIVE; 


(* Assure consistent translations *) 
Flush_linear_context; 


Flags Affected 
RFLAGS.TF is restored from the value previously saved in EENTER or ERESUME. 


Protected Mode Exceptions 
#GP(0) If executed outside an enclave. 

If RBX is outside the CS segment. 
#PF(error code) If a page fault occurs in accessing memory. 


64-Bit Mode Exceptions 
#GP(0) If executed outside an enclave. 
If RBX is not canonical. 
#PF(error code) If a page fault occurs in accessing memory operands. 
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EGETKEY—Retrieves a Cryptographic Key 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 01H IR VIV SGX1 This leaf function retrieves a cryptographic key. 
ENCLU[EGETKEY] 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EGETKEY (In) Return error code (Out) | Address to a KEYREQUEST (In) | Address of the OUTPUTDATA (In) 
Description 


The ENCLU[EGETKEY] instruction returns a 128-bit secret key from the processor specific key hierarchy. The 
register RBX contains the effective address of a KEYREQUEST structure, which the instruction interprets to deter- 
mine the key being requested. The Requesting Keys section below provides a description of the keys that can be 
requested. The RCX register contains the effective address where the key will be returned. Both the addresses in 
RBX & RCX should be locations inside the enclave. 


EGETKEY derives keys using a processor unique value to create a specific key based on a number of possible inputs. 
This instruction leaf can only be executed inside an enclave. 


EEGETKEY Memory Parameter Semantics 


KEYREQUEST OUTPUTDATA 


Enclave read access Enclave write access 


After validating the operands, the instruction determines which key is to be produced and performs the following 
actions: 


¢ The instruction assembles the derivation data for the key based on the Table 36-64. 
* Computes derived key using the derivation data and package specific value. 
* Outputs the calculated key to the address in RCX. 


The instruction fails with #GP(0) if the operands are not properly aligned. Successful completion of the instruction 
will clear RFLAGS.{ZF, CF, AF, OF, SF, PF}. The instruction returns an error code if the user tries to request a key 
based on an invalid CPUSVN or ISVSVN (when the user request is accepted, see the table below), requests a key 
for which it has not been granted the attribute to request, or requests a key that is not supported by the hardware. 
These checks may be performed in any order. Thus, an indication by error number of one cause (for example, 
invalid attribute) does not imply that there are not also other errors. Different processors may thus give different 
error numbers for the same Enclave. The correctness of software should not rely on the order resulting from the 
checks documented in this section. In such cases the ZF flag is set and the corresponding error bit 
(SGX_INVALID_SVN, SGX_INVALID_ATTRIBUTE, SGX_INVALID_KEYNAME) is set in RAX and the data at the 
address specified by RCX is unmodified. 


Requesting Keys 


The KEYREQUEST structure (see Section 33.18.1) identifies the key to be provided. The Keyrequest.KeyName field 
identifies which type of key is requested. 


Deriving Keys 


Key derivation is based on a combination of the enclave specific values (See Table 36-64) and a processor key. 
Depending on the key being requested a field may either be included by definition or the value may be included 
from the KeyRequest. A “yes” in Table 36-64 indicates the value for the field is included from its default location, 
identified in the source row, and a “request” indicates the values for the field is included from its corresponding 
KeyRequest field. 
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Table 36-64. Key Derivation 


Owner | CPU ISvV ISVEXT | ISVFAM CONFIG | CONFIGS 
Key Name | Attributes Epoch | SVN ISV SVN | PRODID | PRODID | ILYID MRENCLAVE | MRSIGNER | ID VN RAND 
Key Y= CR_SGX | Y= R= SECS. SECS.IS | SECS.IS | SECS. SECS. SECS.CO | SECS.CO | Req. 
Dependent | SECS.ATTRIBUTES | OWNER | CPUSVN | ReqISV | ISVID VEXTPR | VFAMIL | MRENCLAVE | MRSIGNER | NFIGID | NFIGSVN | KEYID 
Constant | and EPOCH | Register; | SVN; ODID YID 
SECS.MISCSELECT 
and 
SECS.CET_ATTRIB 
UTES; 
Source 
R:= AttribMask & Ri= 
SECS.ATTRIBUTES Req.CPU 
and SVN; 
SECS.MISCSELECT 
and 
SECS.CET_ATTRIB 
UTES; 
EINITTOKEN | Yes Request Yes Request | Request | Yes No No No Yes No No Request 
Report Yes Yes Yes Yes No No No No Yes No Yes Yes Request 
Seal Yes Request Yes Request | Request | Request | Request | Request | Request Request Request | Request | Request 
Provisioning | Yes Request No Request | Request | Yes No No No Yes No No Yes 
Provisioning | Yes Request No Request | Request | Request | Request | Request | No Yes Request | Request | Yes 
Seal 


Keys that permit the specification of a CPU or ISV's code's, or enclave configuration's SVNs have additional require- 
ments. The caller may not request a key for an SVN beyond the current CPU, ISV or enclave configuration's SVN, 
respectively. 


Several keys are access controlled. Access to the Provisioning Key and Provisioning Seal key requires the enclave's 
ATTRIBUTES.PROVISIONKEY be set. The EINITTOKEN Key requires ATTRIBUTES.EINITTOKEN_KEY be set and 
SECS.MRSIGNER equal IA32_SGXLEPUBKEYHASH. 


Some keys are derived based on a hardcode PKCS padding constant (352 byte string): 
HARDCODED_PKCS1_5_ PADDING[15:0] := 0100H; 

HARDCODED_PKCS1_5_PADDING[2655:16] := SignExtend330Byte(-1); // 330 bytes of OFFH 
HARDCODED_PKCS1_5_ PADDING[2815:2656] := 2004000501020403650148866009060D30313000H; 


The error codes are: 


Table 36-65. EGETKEY Return Value in RAX 


Error Code (see Table 36-4) Value Description 
No Error 0 EGETKEY successful. 

[SGX_INVALID_ATTRIBUTE =~~—~*|~—~—_ | The KEYREQUEST contains a KEYNAME for which the enclave is not authorized. __| 
SGX_INVALID_CPUSVN If KEYREQUEST.CPUSVN is an unsupported platforms CPUSVN value. 
SGX_INVALID_ISVSVN If KEYREQUEST software SVN (ISVSVN or CONFIGSVN) is greater than the 

enclave'’s corresponding SVN. 
SGX_INVALID_KEYNAME If KEYREQUEST.KEYNAME is an unsupported value. 


Concurrency Restrictions 


Table 36-66. Base Concurrency Restrictions of EGETKEY 


Base Concurrency Restrictions 
Leaf Parameter : - — 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EGETKEY KEYREQUEST [DS:RBX] Concurrent 
OUTPUTDATA [DS:RCX] Concurrent 
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Table 36-67. Additional Concurrency Restrictions of EGETKEY 


Additional Concurrency Restrictions 
Leaf Parameter een feces ae vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EGETKEY KEYREQUEST Concurrent Concurrent Concurrent 
[DS:RBX] 
OUTPUTDATA Concurrent Concurrent Concurrent 
[DS:RCX] 
Operation 
Temp Variables in EGETKEY Operational Flow 
Name Type Size (Bits) | Description 
TMP_CURRENTSECS Address of the SECS for the currently executing enclave. 
TMP_KEYDEPENDENCIES Temp space for key derivation. 
TMP_ATTRIBUTES 128 Temp Space for the calculation of the sealable Attributes. 
TMP_ISVEXTPRODID 16 bytes Temp Space for ISVEXTPRODID. 
TMP_ISVPRODID 2 bytes Temp Space for ISVPRODID. 
TMP_ISVFAMILYID 16 bytes Temp Space for ISVFAMILYID. 
TMP_CONFIGID 64 bytes Temp Space for CONFIGID. 
TMP_CONFIGSVN 2 bytes Temp Space for CONFIGSVN. 
TMP_OUTPUTKEY 128 Temp Space for the calculation of the key. 


(* Make sure KEYREQUEST is properly aligned and inside the current enclave *) 
IF ( (DS:RBX is not 512Byte aligned) or (DS:RBX is within CR_ELRANGE) ) 
THEN #GP(0); Fl; 


(* Make sure DS:RBX is an EPC address and the EPC page is valid *) 
IF ((DS:RBX does not resolve to an EPC address) or (EPCM(DS:RBX).VALID = 0) ) 
THEN #PF(DS:RBX); FI; 


IF (EPCM(DS:RBX).BLOCKED = 1) 
THEN #PF(DS:RBX); Fl; 


(* Check page parameters for correctness *) 

IF ( (EPCM(DS:RBX).PT # PT_REG) or (EPCM(DS:RBX).ENCLAVESECS # CR_ACTIVE_SECS) or (EPCM(DS:RBX).PENDING = 1) or 
(EPCM(DS:RBX).MODIFIED = 1) or (EPCM(DS:RBX).ENCLAVEADDRESS # (DS:RBX & ~OFFFH) ) or (EPCM(DS:RBX).R = 0) ) 
THEN #PF(DS:RBX); 

Fl; 


(* Make sure OUTPUTDATA is properly aligned and inside the current enclave *) 
IF ( (DS:RCX is not 16Byte aligned) or (DS:RCX is not within CR_ELRANGE) ) 
THEN #GP(0); Fl; 


(* Make sure DS:RCX is an EPC address and the EPC page is valid *) 
IF ((DS:RCX does not resolve to an EPC address) or (EPCM(DS:RCX).VALID = 0) ) 
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THEN #PF(DS:RCX); Fl; 


IF (EPCM(DS:RCX).BLOCKED = 1) 
THEN #PF(DS:RCX); Fl; 


(* Check page parameters for correctness *) 

IF ((EPCM(DS:RCX).PT # PT_REG) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) or (EPCM(DS:RCX).PENDING = 1) or 
(EPCM(DS:RCX).MODIFIED = 1) or (EPCM(DS:RCX).ENCLAVEADDRESS # (DS:RCX & ~OFFFH) ) or (EPCM(DS:RCX).W = 0) ) 
THEN #PF(DS:RCX); 

Fl; 


(* Verify RESERVED spaces in KEYREQUEST are valid *) 
IF ((DS:RBX).RESERVED # 0) or (DS:RBX.KEYPOLICY.RESERVED # 0) ) 
THEN #GP(0); FI; 


TMP_CURRENTSECS := CR_ACTIVE_SECS; 


(* Verify that CONFIGSVN & New Policy bits are not used if KSS is not enabled *) 
IF ((TMP_CURRENTSECS.ATTRIBUTES.KSS == 0) AND ((DS:RBX.KEYPOLICY & Ox003C # 0) OR (DS:RBX.CONFIGSVN > 0))) 
THEN #GP(0); Fl; 
(* Determine which enclave attributes that must be included in the key. Attributes that must always be include INIT & DEBUG *) 
REQUIRED_SEALING_MASK[1 27:0] := QO000000 00000000 00000000 00000003H; 
TMP_ATTRIBUTES := (DS:RBX.ATTRIBUTEMASK | REQUIRED_SEALING_MASK) & TMP_CURRENTSECS.ATTRIBUTES; 


(* Compute MISCSELECT fields to be included *) 
TMP_MISCSELECT := DS:RBX.MISCMASK & TMP_CURRENTSECS.MISCSELECT 


(* Compute CET_ATTRIBUTES fields to be included *) 
IF (CPUID.(EAX=1 2H, ECX=1):E€AX[6] = 1) 

THEN TMP_CET_ATTRIBUTES := DS:RBX.CET_ATTRIBUTES_ MASK & TMP_CURRENTSECS.CET_ATTRIBUTES; Fl; 
TMP_KEYDEPENDENCIES := 0; 


CASE (DS:RBX.KEYNAME) 
SEAL_KEY: 
IF (DS:RBX.CPUSVN is beyond current CPU configuration) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_CPUSVN; 
GOTO EXIT; 
Fl; 
IF (DS:RBX.ISVSVN > TMP_CURRENTSECS.ISVSVN) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ISVSVN; 
GOTO EXIT; 
Fl; 
IF (DS:RBX.CONFIGSVN > TMP_CURRENTSECS.CONFIGSVN) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ISVSVN; 
GOTO EXIT; 
Fl; 


(*Include enclave identity?*) 
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TMP_MRENCLAVE := 0; 
IF (DS:RBX.KEYPOLICY.MRENCLAVE = 1) 
THEN TMP_MRENCLAVE := TMP_CURRENTSECS.MRENCLAVE; 
FI; 
(*Include enclave author?*) 
TMP_MRSIGNER := 0; 
IF (DS:RBX.KEYPOLICY.MRSIGNER = 1) 
THEN TMP_MRSIGNER := TMP_CURRENTSECS.MRSIGNER; 
FI; 
(* Include enclave product family ID? *) 
TMP_ISVFAMILYID := 0; 
IF (DS:RBX.KEYPOLICY.ISVFAMILYID = 1) 
THEN TMP_ISVFAMILYID := TMP_CURRENTSECS.ISVFAMILYID; 
FI; 


(* Include enclave product ID? *) 
TMP_ISVPRODID := 0; 
IF (DS:RBX.KEYPOLICY.NOISVPRODID = 0) 
TMP_ISVPRODID := TMP_CURRENTSECS.ISVPRODID; 
FI; 


(* Include enclave Config ID? *) 

TMP_CONFIGID := 0; 

TMP_CONFIGSVN := 0; 

IF (DS:RBX.KEYPOLICY.CONFIGID = 1) 
TMP_CONFIGID := TMP_CURRENTSECS.CONFIGID; 
TMP_CONFIGSVN := DS:RBX.CONFIGSVN; 

FI; 


(* Include enclave extended product ID? *) 

TMP_ISVEXTPRODID := 0; 

IF (DS:RBX.KEYPOLICY.ISVEXTPRODID = 1 ) 
TMP_ISVEXTPRODID := TMP_CURRENTSECS.ISVEXTPRODID; 

FI; 


//Determine values key is based on 
TMP_KEYDEPENDENCIES.KEYNAME := SEAL_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := TMP_ISVFAMILYID; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := TMP_ISVEXTPRODID; 
TMP_KEYDEPENDENCIES.ISVPRODID := TMP_ISVPRODID; 
TMP_KEYDEPENDENCIES.ISVSVN := DS:RBX.ISVSVN; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := CR_SGXOWNEREPOCH; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := TMP_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := DS:RBX.ATTRIBUTEMASK; 
TMP_KEYDEPENDENCIES.MRENCLAVE := TMP_MRENCLAVE; 
TMP_KEYDEPENDENCIES.MRSIGNER := TMP_MRSIGNER; 
TMP_KEYDEPENDENCIES.KEYID := DS:RBX.KEYID; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := CR_SEAL_FUSES; 
TMP_KEYDEPENDENCIES.CPUSVN := DS:RBX.CPUSVN; 
TMP_KEYDEPENDENCIES.PADDING := TMP_CURRENTSECS.PADDING; 
TMP_KEYDEPENDENCIES.MISCSELECT := TMP_MISCSELECT; 
TMP_KEYDEPENDENCIES.MISCMASK := ~DS:RBX.MISCMASK; 
TMP_KEYDEPENDENCIES.KEYPOLICY := DS:RBX.KEYPOLICY; 
TMP_KEYDEPENDENCIES.CONFIGID := TMP_CONFIGID; 
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TMP_KEYDEPENDENCIES.CONFIGSVN := TMP_CONFIGSVN; 
IF CPUID.(EAX=12H, ECX=1):E€AX[6] = 1 
THEN 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := TMP_CET_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES _MASK := DS:RBX.CET_ATTRIBUTES _MASK; 
Fl; 
BREAK; 
REPORT_KEY: 
//Determine values key is based on 
TMP_KEYDEPENDENCIES.KEYNAME := REPORT_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := 0; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVSVN := 0; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := CR_SGXOWNEREPOCH; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := TMP_CURRENTSECS.ATTRIBUTES; 
TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := 0; 
TMP_KEYDEPENDENCIES.MRENCLAVE := TMP_CURRENTSECS.MRENCLAVE; 
TMP_KEYDEPENDENCIES.MRSIGNER := 0; 
TMP_KEYDEPENDENCIES.KEYID := DS:RBX.KEYID; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := CR_SEAL_FUSES; 
TMP_KEYDEPENDENCIES.CPUSVN := CR_CPUSVN; 
TMP_KEYDEPENDENCIES.PADDING := HARDCODED_PKCS1_5_ PADDING; 
TMP_KEYDEPENDENCIES.MISCSELECT := TMP_CURRENTSECS.MISCSELECT; 
TMP_KEYDEPENDENCIES.MISCMASK := 0; 
TMP_KEYDEPENDENCIES.KEYPOLICY := 0; 
TMP_KEYDEPENDENCIES.CONFIGID := TMP_CURRENTSECS.COMFIGID; 
TMP_KEYDEPENDENCIES.CONFIGSVN := TMP_CURRENTSECS.CONFIGSVN; 
IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
THEN 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := TMP_CURRENTSECS.CET_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES_MASK := 0; 
Fl; 
BREAK; 
EINITTOKEN_KEY: 
(* Check ENCLAVE has EINITTOKEN Key capability *) 
IF (TMP_CURRENTSECS.ATTRIBUTES.EINITTOKEN_KEY = 0) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO ExIT; 
Fl; 
IF (DS:RBX.CPUSVN is beyond current CPU configuration) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_CPUSVN; 
GOTO ExIT; 
Fl; 
IF (DS:RBX.ISVSVN > TMP_CURRENTSECS.ISVSVN) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ISVSVN; 
GOTO ExIT; 
Fl; 
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(* Determine values key is based on *) 
TMP_KEYDEPENDENCIES.KEYNAME := EINITTOKEN_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := 0; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVPRODID := TMP_CURRENTSECS.ISVPRODID 
TMP_KEYDEPENDENCIES.ISVSVN := DS:RBX.ISVSVN; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := CR_SGXOWNEREPOCH; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := TMP_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := 0; 
TMP_KEYDEPENDENCIES.MRENCLAVE := 0; 
TMP_KEYDEPENDENCIES.MRSIGNER := TMP_CURRENTSECS.MRSIGNER; 
TMP_KEYDEPENDENCIES.KEYID := DS:RBX.KEYID; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := CR_SEAL_FUSES; 
TMP_KEYDEPENDENCIES.CPUSVN := DS:RBX.CPUSVN; 
TMP_KEYDEPENDENCIES.PADDING := TMP_CURRENTSECS.PADDING; 
TMP_KEYDEPENDENCIES.MISCSELECT := TMP_MISCSELECT; 
TMP_KEYDEPENDENCIES.MISCMASK := 0; 
TMP_KEYDEPENDENCIES.KEYPOLICY := 0; 
TMP_KEYDEPENDENCIES.CONFIGID := 0; 
TMP_KEYDEPENDENCIES.CONFIGSVN := 0; 
IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
THEN 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := TMP_CET_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES _MASK := 0; 
FI; 
BREAK; 
PROVISION_KEY: 
(* Check ENCLAVE has PROVISIONING capability *) 
IF (TMP_CURRENTSECS.ATTRIBUTES.PROVISIONKEY = 0) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO EXIT; 
FI; 
IF (DS:RBX.CPUSVN is beyond current CPU configuration) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_CPUSVN; 
GOTO EXIT; 
FI; 
IF (DS:RBX.ISVSVN > TMP_CURRENTSECS.ISVSVN) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ISVSVN; 
GOTO EXIT; 
FI; 
(* Determine values key is based on *) 
TMP_KEYDEPENDENCIES.KEYNAME := PROVISION_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := 0; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVPRODID := TMP_CURRENTSECS.ISVPRODID; 
TMP_KEYDEPENDENCIES.ISVSVN := DS:RBX.ISVSVN; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := 0; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := TMP_ATTRIBUTES; 
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TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := DS:RBX.ATTRIBUTEMASK; 
TMP_KEYDEPENDENCIES.MRENCLAVE := 0; 
TMP_KEYDEPENDENCIES.MRSIGNER := TMP_CURRENTSECS.MRSIGNER; 
TMP_KEYDEPENDENCIES.KEYID := 0; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := 0; 
TMP_KEYDEPENDENCIES.CPUSVN := DS:RBX.CPUSVN; 
TMP_KEYDEPENDENCIES.PADDING := TMP_CURRENTSECS.PADDING; 
TMP_KEYDEPENDENCIES.MISCSELECT := TMP_MISCSELECT; 
TMP_KEYDEPENDENCIES.MISCMASK := ~DS:RBX.MISCMASK; 
TMP_KEYDEPENDENCIES.KEYPOLICY := 0; 
TMP_KEYDEPENDENCIES.CONFIGID := 0; 
IF (CPUID.(EAX=1 2H, ECX=1):E€AX[6] = 1) 
THEN 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := TMP_CET_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES _MASK := 0; 
Fl; 
BREAK; 
PROVISION_SEAL_KEY: 
(* Check ENCLAVE has PROVISIONING capability *) 
IF (TMP_CURRENTSECS.ATTRIBUTES.PROVISIONKEY = 0) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ATTRIBUTE; 
GOTO EXIT; 
Fl; 
IF (DS:RBX.CPUSVN is beyond current CPU configuration) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_CPUSVN; 
GOTO EXIT; 
Fl; 
IF (DS:RBX.ISVSVN > TMP_CURRENTSECS.ISVSVN) 
THEN 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_ISVSVN; 
GOTO EXIT; 
Fl; 
(* Include enclave product family ID? *) 
TMP_ISVFAMILYID := 0; 
IF (DS:RBX.KEYPOLICY.ISVFAMILYID = 1) 
THEN TMP_ISVFAMILYID := TMP_CURRENTSECS.ISVFAMILYID; 
Fl; 


(* Include enclave product ID? *) 
TMP_ISVPRODID := 0; 
IF (DS:RBX.KEYPOLICY.NOISVPRODID = 0) 
TMP_ISVPRODID := TMP_CURRENTSECS.ISVPRODID; 
Fl; 


(* Include enclave Config ID? *) 

TMP_COMFIGID := 0; 

TMP_CONFIGSVN := 0; 

IF (DS:RBX.KEYPOLICY.CONFIGID = 1) 
TMP_CONFIGID := TMP_CURRENTSECS.CONFIGID; 
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TMP_CONFIGSVN := DS:RBX.CONFIGSVN; 
Fl; 


(* Include enclave extended product ID? *) 

TMP_ISVEXTPRODID := 0; 

IF (DS:RBX.KEYPOLICY.ISVEXTPRODID = 1) 
TMP_ISVEXTPRODID := TMP_CURRENTSECS.ISVEXTPRODID; 

FI; 


(* Determine values key is based on *) 
TMP_KEYDEPENDENCIES.KEYNAME := PROVISION_SEAL_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := TMP_ISVFAMILYID; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := TMP_ISVEXTPRODID; 
TMP_KEYDEPENDENCIES.ISVPRODID := TMP_ISVPRODID; 
TMP_KEYDEPENDENCIES.ISVSVN := DS:RBX.ISVSVN; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := 0; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := TMP_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := DS:RBX.ATTRIBUTEMASK; 
TMP_KEYDEPENDENCIES.MRENCLAVE := 0; 
TMP_KEYDEPENDENCIES.MRSIGNER := TMP_CURRENTSECS.MRSIGNER; 
TMP_KEYDEPENDENCIES.KEYID := 0; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := CR_SEAL_FUSES; 
TMP_KEYDEPENDENCIES.CPUSVN := DS:RBX.CPUSVN; 
TMP_KEYDEPENDENCIES.PADDING := TMP_CURRENTSECS.PADDING; 
TMP_KEYDEPENDENCIES.MISCSELECT := TMP_MISCSELECT; 
TMP_KEYDEPENDENCIES.MISCMASK := ~DS:RBX.MISCMASK; 
TMP_KEYDEPENDENCIES.KEYPOLICY := DS:RBX.KEYPOLICY; 
TMP_KEYDEPENDENCIES.CONFIGID := TMP_CONFIGID; 
TMP_KEYDEPENDENCIES.CONFIGSVN := TMP_CONFIGSVN; 
IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
THEN 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := TMP_CET_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES _MASK := 0; 
FI; 
BREAK; 
DEFAULT: 
(* The value of KEYNAME is invalid *) 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_KEYNAME; 
GOTO EXIT: 
ESAC; 


(* Calculate the final derived key and output to the address in RCX *) 
TMP_OUTPUTKEY := derivekey(TMP_KEYDEPENDENCIES); 

DS:RCX[1 5:0] := TMP_OUTPUTKEY; 

RAX := 0; 

RFLAGS.ZF := 0; 


EXIT: 

RFLAGS.CF := 0; 
RFLAGS.PF := 0; 
RFLAGS.AF := 0; 
RFLAGS.OF := 0; 
RFLAGS.SF := 0; 
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ZF is cleared if successful, otherwise ZF is set. CF, PF, AF, OF, SF are cleared. 


Protected Mode Exceptions 


#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If a memory operand effective address is outside the current enclave. 
If an effective address is not properly aligned. 

If an effective address is outside the DS segment limit. 

If KEYREQUEST format is invalid. 

If a page fault occurs in accessing memory. 


64-Bit Mode Exceptions 


#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If a memory operand effective address is outside the current enclave. 
If an effective address is not properly aligned. 

If an effective address is not canonical. 

If KEYREQUEST format is invalid. 

If a page fault occurs in accessing memory operands. 
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EMODPE—Extend an EPC Page Permissions 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = O6H IR VIN SGX2 This leaf function extends the access rights of an existing EPC 
ENCLU[EMODPE] page. 


Instruction Operand Encoding 


Op/En EAX RBX RCX 
IR EMODPE (In) Address of a SECINFO (In) Address of the destination EPC page (In) 
Description 


This leaf function extends the access rights associated with an existing EPC page in the running enclave. THE RWX 
bits of the SECINFO parameter are treated as a permissions mask; supplying a value that does not extend the page 
permissions will have no effect. This instruction leaf can only be executed when inside the enclave. 


RBX contains the effective address of a SECINFO structure while RCX contains the effective address of an EPC page. 
The table below provides additional information on the memory parameter of the EMODPE leaf function. 


EMODPE Memory Parameter Semantics 


SECINFO EPCPAGE 


Read access permitted by Non Enclave Read access permitted by Enclave 


The instruction faults if any of the following: 


EMODPE Faulting Conditions 
The operands are not properly aligned. If security attributes of the SECINFO page make the page inaccessible. 
The EPC page is locked by another thread. RBX does not contain an effective address in an EPC page in the running enclave. 
The EPC page is not valid. RCX does not contain an effective address of an EPC page in the running enclave. 
SECINFO contains an invalid request. 


Concurrency Restrictions 


Table 36-68. Base Concurrency Restrictions of EMODPE 


Base Concurrency Restrictions 
Leaf Parameter - - ——S 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EMODPE Target [DS:RCX] Concurrent 
SECINFO [DS:RBX] Concurrent 


Table 36-69. Additional Concurrency Restrictions of EMODPE 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EMODPE Target [DS:RCX] Exclusive #GP Concurrent Concurrent 
SECINFO [DS:RBX] Concurrent Concurrent Concurrent 
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Operation 
Temp Variables in EMODPE Operational Flow 
Name Type Size (bits) | Description 
SCRATCH_SECINFO SECINFO 512 Scratch storage for holding the contents of DS:RBX. 


IF (DS:RBX is not 64Byte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RCX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF ((DS:RBX is not within CR_ELRANGE) or (DS:RCX is not within CR_ELRANGE) ) 
THEN #GP(0); Fl; 


IF (DS:RBX does not resolve within an EPC) 
THEN #PF(DS:RBX); FI; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


IF ( (EPCM(DS:RBX).VALID = 0) or (EPCM(DS:RBX).R = 0) or (EPCM(DS:RBX).PENDING # 0) or (EPCM(DS:RBX).MODIFIED # 0) or 
(EPCM(DS:RBX).BLOCKED # 0) or (EPCM(DS:RBX).PT # PT_REG) or (EPCM(DS:RBX).ENCLAVESECS # CR_ACTIVE_SECS) or 
(EPCM(DS:RBX).ENCLAVEADDRESS # (DS:RBX & ~OxFFF)) ) 

THEN #PF(DS:RBX); Fl; 


SCRATCH_SECINFO := DS:RBX; 


(* Check for misconfigured SECINFO flags*) 
IF (SCRATCH_SECINFO reserved fields are not zero ) 
THEN #GP(0); Fl; 


(* Check security attributes of the EPC page *) 

IF ((EPCM(DS:RCX).VALID = 0) or (EPCM(DS:RCX).PENDING # 0) or (EPCM(DS:RCX).MODIFIED # 0) or 
(EPCM(DS:RCX).BLOCKED # 0) or (EPCM(DS:RCX).PT # PT_REG) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) ) 
THEN #PF(DS:RCX); Fl; 


(* Check the EPC page for concurrency *) 
IF (EPC page in use by another SGX2 instruction) 
THEN #GP(0); Fl; 


(* Re-Check security attributes of the EPC page *) 

IF ((EPCM(DS:RCX).VALID = 0) or (EPCM(DS:RCX).PENDING # 0) or (EPCM(DS:RCX).MODIFIED # 0) or 
(EPCM(DS:RCX).PT # PT_REG) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) or 
(EPCM(DS:RCX).ENCLAVEADDRESS # DS:RCX)) 

THEN #PF(DS:RCX); Fl; 


(* Check for misconfigured SECINFO flags*) 


IF ( (EPCM(DS:RCX).R = 0) and (SCRATCH_SECINFO.FLAGS.R = 0) and (SCRATCH_SECINFO.FLAGS.W # 0) ) 
THEN #GP(0); Fl; 
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(* Update EPCM permissions *) 

EPCM(DS:RCX).R := EPCM(DS:RCX).R | SCRATCH_SECINFO.FLAGS.R; 
EPCM(DS:RCX).W := EPCM(DS:RCX).W | SCRATCH_SECINFO.FLAGS.W; 
EPCM(DS:RCX).X := EPCM(DS:RCX).X | SCRATCH_SECINFO.FLAGS.X; 


Flags Affected 
None 


Protected Mode Exceptions 

#GP(0) If executed outside an enclave. 
If a memory operand effective address is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 

#PF(error code) If a page fault occurs in accessing memory operands. 


64-Bit Mode Exceptions 
#GP(0) If executed outside an enclave. 
If a memory operand is non-canonical form. 
If a memory operand is not properly aligned. 
If a memory operand is locked. 
#PF(error code) If a page fault occurs in accessing memory operands. 
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EREPORT—Create a Cryptographic Report of the Enclave 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = OOH IR VIV SGX1 This leaf function creates a cryptographic report of the enclave. 
ENCLU[EREPORT] 
Instruction Operand Encoding 
Op/En EAX RBX RCX RDX 
IR EREPORT (In) Address of TARGETINFO | Address of REPORTDATA Address where the REPORT is 
(In) (In) written to in an OUTPUTDATA (In) 
Description 


This leaf function creates a cryptographic REPORT that describes the contents of the enclave. This instruction leaf 
can only be executed when inside the enclave. The cryptographic report can be used by other enclaves to deter- 
mine that the enclave is running on the same platform. 


RBX contains the effective address of the MRENCLAVE value of the enclave that will authenticate the REPORT 
output, using the REPORT key delivered by EGETKEY command for that enclave. RCX contains the effective address 
of a 64-byte REPORTDATA structure, which allows the caller of the instruction to associate data with the enclave 
from which the instruction is called. RDX contains the address where the REPORT will be output by the instruction. 


EREPORT Memory Parameter Semantics 
REPORTDATA 


Read access by Enclave 


TARGETINFO 


Read access by Enclave 


OUTPUTDATA 


Read/Write access by Enclave 


This instruction leaf perform the following: 
1. Validate the 3 operands (RBX, RCX, RDX) are inside the enclave. 
2. Compute a report key for the target enclave, as indicated by the value located in RBX(TARGETINFO). 


3. Assemble the enclave SECS data to complete the REPORT structure (including the data provided using the RCX 
(REPORTDATA) operand). 


4. Computes a cryptographic hash over REPORT structure. 

5. Add the computed hash to the REPORT structure. 

6. Output the completed REPORT structure to the address in RDX (OUTPUTDATA). 
The instruction fails if the operands are not properly aligned. 


CR_REPORT_KEYID, used to provide key wearout protection, is populated with a statistically unique value on boot 
of the platform by a trusted entity within the SGX TCB. 


The instruction faults if any of the following: 


EREPORT Faulting Conditions 
An memory address does not resolve in an EPC page. 
If the EPC page is blocked. 


An effective address not properly aligned. 


If accessing an invalid EPC page. 


May page fault. 
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Concurrency Restrictions 


Table 36-70. Base Concurrency Restrictions of EREPORT 


teat pararieiee Base oe Restrictions _ 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
EREPORT TARGETINFO [DS:RBX] Concurrent 
REPORTDATA [DS:RCX] Concurrent 
OUTPUTDATA [DS:RDX] Concurrent 
Table 36-71. Additional Concurrency Restrictions of EREPORT 
Additional Concurrency Restrictions 
Leaf Parameter ten Tbe: car: vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EREPORT TARGETINFO [DS:RBX] | Concurrent Concurrent Concurrent 
REPORTDATA Concurrent Concurrent Concurrent 
[DS:RCX] 
OUTPUTDATA Concurrent Concurrent Concurrent 
[DS:RDX] 
Operation 
Temp Variables in EREPORT Operational Flow 
Name Type Size (bits) | Description 
TMP_ATTRIBUTES 32 Physical address of SECS of the enclave to which source operand belongs. 
TMP_CURRENTSECS Address of the SECS for the currently executing enclave. 
TMP_KEYDEPENDENCIES Temp space for key derivation. 
TMP_REPORTKEY 128 REPORTKEY generated by the instruction. 
TMP_REPORT 3712 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1); 


(* Address verification for TARGETINFO (RBX) *) 
IF ( (DS:RBX is not 512Byte Aligned) or (DS:RBX is not within CR_ELRANGE) ) 
THEN #GP(0); Fl; 


IF (DS:RBX does not resolve within an EPC) 
THEN #PF(DS:RBX); FI; 


IF (EPCM(DS:RBX).VALID = 0) 
THEN #PF(DS:RBX); Fl; 


IF (EPCM(DS:RBX).BLOCKED = 1) 
THEN #PF(DS:RBX); Fl; 


(* Check page parameters for correctness *) 


IF ( (EPCM(DS:RBX).PT # PT_REG) or (EPCM(DS:RBX).ENCLAVESECS # CR_ACTIVE_SECS) or (EPCM(DS:RBX).PENDING = 1) or 
(EPCM(DS:RBX).MODIFIED = 1) or (EPCM(DS:RBX).ENCLAVEADDRESS # (DS:RBX & ~OFFFH) ) or (EPCM(DS:RBX).R = 0) ) 
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THEN #PF(DS:RBX); 
Fl; 


(* Verify RESERVED spaces in TARGETINFO are valid *) 
IF (DS:RBX.RESERVED != 0) 
THEN #GP(0); Fl; 


(* Address verification for REPORTDATA (RCX) *) 
IF ( (DS:RCX is not 128Byte Aligned) or (DS:RCX is not within CR_ELRANGE) ) 
THEN #GP(0); Fl; 


IF (DS:RCX does not resolve within an EPC) 
THEN #PF(DS:RCX); Fl; 


IF (EPCM(DS:RCX).VALID = 0) 
THEN #PF(DS:RCX); Fl; 


IF (EPCM(DS:RCX).BLOCKED = 1) 
THEN #PF(DS:RCX); Fl; 


(* Check page parameters for correctness *) 

IF ((EPCM(DS:RCX).PT # PT_REG) or (EPCM(DS:RCX).ENCLAVESECS # CR_ACTIVE_SECS) or (EPCM(DS:RCX).PENDING = 1) or 
(EPCM(DS:RCX).MODIFIED = 1) or (EPCM(DS:RCX).ENCLAVEADDRESS # (DS:RCX & ~OFFFH) ) or (EPCM(DS:RCX).R = 0) ) 
THEN #PF(DS:RCX); 

Fl; 


(* Address verification for OUTPUTDATA (RDX) *) 
IF ( (DS:RDX is not 512Byte Aligned) or (DS:RDX is not within CR_ELRANGE) ) 
THEN #GP(0); Fl; 


IF (DS:RDX does not resolve within an EPC) 
THEN #PF(DS:RDX); FI; 


IF (EPCM(DS:RDX).VALID = 0) 
THEN #PF(DS:RDX); Fl; 


IF (EPCM(DS:RDX).BLOCKED = 1) 
THEN #PF(DS:RDX); Fl: 


(* Check page parameters for correctness *) 

IF ((EPCM(DS:RDX).PT # PT_REG) or (EPCM(DS:RDX).ENCLAVESECS # CR_ACTIVE_SECS) or (EPCM(DS:RCX).PENDING = 1) or 
(EPCM(DS:RCX).MODIFIED = 1) or (EPCM(DS:RDX).ENCLAVEADDRESS # (DS:RDX & ~OFFFH) ) or (EPCM(DS:RDX).W = 0) ) 
THEN #PF(DS:RDX); 

Fl; 


(* REPORT MAC needs to be computed over data which cannot be modified *) 
TMP_REPORT.CPUSVN := CR_CPUSVN; 

TMP_REPORT.ISVFAMILYID := TMP_CURRENTSECS.ISVFAMILYID; 
TMP_REPORT.ISVEXTPRODID := TMP_CURRENTSECS.ISVEXTPRODID; 
TMP_REPORT.ISVPRODID := TMP_CURRENTSECS.ISVPRODID; 
TMP_REPORT.ISVSVN := TMP_CURRENTSECS.ISVSVN; 
TMP_REPORT.ATTRIBUTES := TMP_CURRENTSECS.ATTRIBUTES; 
TMP_REPORT.REPORTDATA := DS:RCX[51 1:0]; 

TMP_REPORT.MRENCLAVE := TMP_CURRENTSECS.MRENCLAVE; 
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TMP_REPORT.MRSIGNER := TMP_CURRENTSECS.MRSIGNER; 
TMP_REPORT.MRRESERVED := 0; 
TMP_REPORT.KEYID[255:0] := CR_REPORT_KEYID; 
TMP_REPORT.MISCSELECT := TMP_CURRENTSECS.MISCSELECT; 
TMP_REPORT.CONFIGID := TMP_CURRENTSECS.CONFIGID; 
TMP_REPORT.CONFIGSVN := TMP_CURRENTSECS.CONFIGSVN; 
IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
THEN TMP_REPORT.CET_ATTRIBUTES := TMP_CURRENTSECS.CET_ATTRIBUTES; Fl; 


(* Derive the report key *) 
TMP_KEYDEPENDENCIES.KEYNAME := REPORT_KEY; 
TMP_KEYDEPENDENCIES.ISVFAMILYID := 0; 
TMP_KEYDEPENDENCIES.ISVEXTPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVPRODID := 0; 
TMP_KEYDEPENDENCIES.ISVSVN := 0; 
TMP_KEYDEPENDENCIES.SGXOWNEREPOCH := CR_SGXOWNEREPOCH; 
TMP_KEYDEPENDENCIES.ATTRIBUTES := DS:RBX.ATTRIBUTES; 
TMP_KEYDEPENDENCIES.ATTRIBUTESMASK := 0; 
TMP_KEYDEPENDENCIES.MRENCLAVE := DS:RBX.MEASUREMENT; 
TMP_KEYDEPENDENCIES.MRSIGNER := 0; 
TMP_KEYDEPENDENCIES.KEYID := TMP_REPORT.KEYID; 
TMP_KEYDEPENDENCIES.SEAL_KEY_FUSES := CR_SEAL_FUSES; 
TMP_KEYDEPENDENCIES.CPUSVN := CR_CPUSVN; 
TMP_KEYDEPENDENCIES.PADDING := TMP_CURRENTSECS.PADDING; 
TMP_KEYDEPENDENCIES.MISCSELECT := DS:RBX.MISCSELECT; 
TMP_KEYDEPENDENCIES.MISCMASK := 0; 
TMP_KEYDEPENDENCIES.KEYPOLICY := 0; 
TMP_KEYDEPENDENCIES.CONFIGID := DS:RBX.CONFIGID; 
TMP_KEYDEPENDENCIES.CONFIGSVN := DS:RBX.CONFIGSVN; 
IF (CPUID.(EAX=1 2H, ECX=1):EAX[6] = 1) 
THEN 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES := DS:RBX.CET_ATTRIBUTES; 
TMP_KEYDEPENDENCIES.CET_ATTRIBUTES _MASK := 0; 
FI; 


(* Calculate the derived key*) 
TMP_REPORTKEY := derivekey(TMP_KEYDEPENDENCIES); 


(* call cryptographic CMAC function, CMAC data are not including MAC&KEYID *) 
TMP_REPORT.MAC := cmac(TMP_REPORTKEY, TMP_REPORT[307 1:0] ); 
DS:RDX[3455: 0] := TMP_REPORT; 


Flags Affected 
None 


Protected Mode Exceptions 
#GP(0) If executed outside an enclave. 
If the address in RCS is outside the DS segment limit. 
If a memory operand is not properly aligned. 
If a memory operand is not in the current enclave. 
#PF(error code) If a page fault occurs in accessing memory operands. 
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#GP(0) 


#PF(error code) 


If executed outside an enclave. 

If RCX is non-canonical form. 

If a memory operand is not properly aligned. 

If a memory operand is not in the current enclave. 


If a page fault occurs in accessing memory operands. 
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ERESUME—Re-Enters an Enclave 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 03H IR VIV SGX1 This leaf function is used to re-enter an enclave after an inter- 
ENCLU[ERESUME] rupt. 


Instruction Operand Encoding 


Op/En RAX RBX RCX 
IR ERESUME (In) Address of a TCS (In) Address of AEP (In) 
Description 


The ENCLU[ERESUME] instruction resumes execution of an enclave that was interrupted due to an exception or 
interrupt, using the machine state previously stored in the SSA. 


ERESUME Memory Parameter Semantics 
TCS 


Enclave read/write access 


The instruction faults if any of the following: 


Address in RBX is not properly aligned. Any TCS.FLAGS's must-be-zero bit is not zero. 

TCS pointed to by RBX is not valid or available or Current 32/64 mode does not match the enclave mode in 

locked. SECS.ATTRIBUTES.MODE64. 

The SECS is in use by another enclave. Either of TCS-specified FS and GS segment is not a subset of the current DS 
segment. 

Any one of DS, ES, CS, SS is not zero. If XSAVE available, CR4.0SXSAVE = 0, but SECS.ATTRIBUTES.XFRM # 3. 

CR4.0SFXSR # 1. If CR4.0SXSAVE = 1, SECS.ATTRIBUTES.XFRM is not a subset of XCRO. 

Offsets 520-535 of the XSAVE area not 0. The bit vector stored at offset 512 of the XSAVE area must be a subset of 
SECS.ATTRIBUTES.XFRM. 

The SSA frame is not valid or in use. 


The following operations are performed by ERESUME: 


¢ RSP and RBP are saved in the current SSA frame on EENTER and are automatically restored on EEXIT or an 
asynchronous exit due to any Interrupt event. 


¢ The AEP contained in RCX is stored into the TCS for use by AEXs.FS and GS (including hidden portions) are 
saved and new values are constructed using TCS.OFSBASE/GSBASE (32 and 64-bit mode) and 
TCS.OFSLIMIT/GSLIMIT (32-bit mode only). The resulting segments must be a subset of the DS segment. 


° If CR4.0SXSAVE == 1, XCRO is saved and replaced by SECS.ATTRIBUTES.XFRM. The effect of RFLAGS.TF 
depends on whether the enclave entry is opt-in or opt-out (see Section 38.1.2): 


— On opt-out entry, TF is saved and cleared (it is restored on EEXIT or AEX). Any attempt to set TF via a POPF 
instruction while inside the enclave clears TF (see Section 38.2.5). 


— On opt-in entry, a single-step debug exception is pended on the instruction boundary immediately after 
EENTER (see Section 38.2.3). 


¢ All code breakpoints that do not overlap with ELRANGE are also suppressed. If the entry is an opt-out entry, all 
code and data breakpoints that overlap with the ELRANGE are suppressed. 
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* On opt-out entry, a number of performance monitoring counters and behaviors are modified or suppressed 
(see Section 38.2.3): 


— All performance monitoring activity on the current thread is suppressed except for incrementing and firing 


of FIXED_CTR1 and FIXED_CTR2. 


— PEBS is suppressed. 


— AnyThread counting on other threads is demoted to MyThread mode and IA32_PERF_GLOBAL_STATUS[60] 
on that thread is set. 


— If the opt-out entry on a hardware thread results in suppression of any performance monitoring, then the 
processor sets IA32_PERF_GLOBAL_STATUS[60] and IA32_PERF_GLOBAL_STATUS[63]. 


Concurrency Restrictions 


Table 36-72. Base Concurrency Restrictions of ERESUME 


lect Saal Base EoneUneney Restrictions _— 
Access On Conflict SGX_CONFLICT VM Exit Qualification 
ERESUME TCS [DS:RBX] Shared #GP 
Table 36-73. Additional Concurrency Restrictions of ERESUME 
Additional Concurrency Restrictions 
Leaf Parameter aicore jens gas vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
ERESUME TCS [DS:RBX] Concurrent Concurrent Concurrent 
Operation 
Temp Variables in ERESUME Operational Flow 
Name Type Size Description 
TMP_FSBASE Effective Address | 32/64 _ | Proposed base address for FS segment. 
TMP_GSBASE Effective Address | 32/64 _ | Proposed base address for FS segment. 
TMP_FSLIMIT Effective Address | 32/64 | Highest legal address in proposed FS segment. 
TMP_GSLIMIT Effective Address | 32/64 | Highest legal address in proposed GS segment. 
TMP_TARGET Effective Address | 32/64 | Address of first instruction inside enclave at which execution is to resume. 
TMP_SECS Effective Address | 32/64 | Physical address of SECS for this enclave. 
TMP_SSA Effective Address | 32/64 | Address of current SSA frame. 
TMP_XSIZE integer 64 Size of XSAVE area based on SECS.ATTRIBUTES.XFRM. 
TMP_SSA_PAGE Effective Address | 32/64 | Pointer used to iterate over the SSA pages in the current frame. 
TMP_GPR Effective Address | 32/64 | Address of the GPR area within the current SSA frame. 
TMP_BRANCH_RECORD | LBR Record From/to addresses to be pushed onto the LBR stack. 


TMP_MODE64 := ((IA32_EFER.LMA = 1) && (CS.L = 1); 


(* Make sure DS is usable, expand up *) 
IF (TMP_MODE64 = 0 and (DS not usable or ( ( DS[S] = 1) and (DS[bit 11] = 0) and DS[bit 10] = 1) ))) 


THEN #GP(0); Fl; 
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(* Check that CS, SS, DS, ES.base is 0 *) 
IF (TMP_MODE64 = 0) 
THEN 

IF(CS.base # 0 or DS.base # 0) #GP(0); FI; 
IF(ES usable and ES.base # 0) #GP(0); FI; 
IF(SS usable and SS.base # 0) #GP(0); FI; 
IF(SS usable and SS.B = 0) #GP(0); FI; 

Fl; 


IF (DS:RBX is not 4KByte Aligned) 
THEN #GP(0); Fl; 


IF (DS:RBX does not resolve within an EPC) 
THEN #PF(DS:RBX); FI; 


(* Check AEP is canonical*) 
IF (TMP_MODE64 = 1 and (CS:RCX is not canonical) ) 
THEN #GP(0); Fl; 


(* Check concurrency of TCS operation*) 
IF (Other Intel SGX instructions is operating on TCS) 
THEN #GP(0); Fl; 


(* TCS verification *) 
IF (EPCM(DS:RBX).VALID = 0) 
THEN #PF(DS:RBX); FI; 


IF (EPCM(DS:RBX).BLOCKED = 1) 
THEN #PF(DS:RBX); Fl; 


IF ((EPCM(DS:RBX).PENDING = 1) or (EPCM(DS:RBX).MODIFIED = 1) 
THEN #PF(DS:RBX); Fl; 


IF ( (EPCM(DS:RBX).ENCLAVEADDRESS # DS:RBX) or (EPCM(DS:RBX).PT # PT_TCS) ) 
THEN #PF(DS:RBX); Fl; 


IF ( (DS:RBX).0SSA is not 4KByte Aligned) 
THEN #GP(0); Fl; 


(* Check proposed FS and GS *) 
IF ( ( (DS:RBX).OFSBASE is not 4KByte Aligned) or ( (DS:RBX).OGSBASE is not 4KByte Aligned) ) 
THEN #GP(0); Fl; 


(* Get the SECS for the enclave in which the TCS resides *) 
TMP_SECS := Address of SECS for TCS; 


(* Make sure that the FLAGS field in the TCS does not have any reserved bits set *) 
IF ( ( (DS:RBX).FLAGS & FFFFFFFFFFFFFFFEH) # 0) 
THEN #GP(0); Fl; 


(* SECS must exist and enclave must have previously been EINITted *) 


IF (the enclave is not already initialized) 
THEN #GP(0); Fl; 
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(* make sure the logical processor's operating mode matches the enclave *) 
IF ((TMP_MODE64 # TMP_SECS.ATTRIBUTES.MODE64BIT) ) 
THEN #GP(0); Fl; 


IF (CR4.OSFXSR = 0) 
THEN #GP(0); Fl; 


(* Check for legal values of SECS.ATTRIBUTES.XFRM *) 
IF (CR4.0SXSAVE = 0) 
THEN 
IF (TMP_SECS.ATTRIBUTES.XFRM # 03H) THEN #GP(0); Fl; 
ELSE 
IF ((TMP_SECS.ATTRIBUTES.XFRM & XCRO) # TMP_SECS.ATTRIBUTES.XFRM) THEN #GP(0); FI; 
FI; 


(* Make sure the SSA contains at least one active frame *) 
IF ( (DS:RBX).CSSA = 0) 
THEN #GP(0); Fl; 


(* Compute linear address of SSA frame *) 
TMP_SSA := (DS:RBX).OSSA + TMP_SECS.BASEADDR + 4096 * TMP_SECS.SSAFRAMESIZE * ( (DS:RBX).CSSA - 1); 
TMP_XSIZE := compute_XSAVE_frame_size(TMP_SECS.ATTRIBUTES.XFRM); 


FOR EACH TMP_SSA_PAGE = TMP_SSA to TMP_SSA + TMP_XSIZE 

(* Check page is read/write accessible *) 

Check that DS:TMP_SSA_PAGE is read/write accessible; 

If a fault occurs, release locks, abort and deliver that fault; 

IF (DS:TMP_SSA_PAGE does not resolve to EPC page) 

THEN #PF(DS:TMP_SSA_PAGE); Fl; 

IF (EPCM(DS:TMP_SSA_PAGE).VALID = 0) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 

IF (EPCM(DS:TMP_SSA_PAGE).BLOCKED = 1) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 

IF ((EPCM(DS:TMP_SSA_PAGE).PENDING = 1) or (EPCM(DS:TMP_SSA_PAGE_.MODIFIED = 1)) 
THEN #PF(DS:TMP_SSA_PAGE); Fl; 

IF (( EPCM(DS:TMP_SSA_PAGE).ENCLAVEADDRESS # DS:TMPSSA_PAGE) or (EPCM(DS:TMP_SSA_PAGE).PT # PT_REG) or 
(EPCM(DS:TMP_SSA_PAGE).ENCLAVESECS # EPCM(DS:RBX).ENCLAVESECS) or 
(EPCM(DS:TMP_SSA_PAGE).R = 0) or (EPCM(DS:TMP_SSA_PAGE).W = 0) ) 

THEN #PF(DS:TMP_SSA_PAGE); Fl; 
CR_XSAVE_PAGE_n := Physical_Address(DS:TMP_SSA_PAGE); 
ENDFOR 


(* Compute address of GPR area”) 
TMP_GPR := TMP_SSA + 4096 * DS:TMP_SECS.SSAFRAMESIZE - sizeof(GPRSGX_AREA); 
Check that DS:TMP_SSA_PAGE is read/write accessible; 
If a fault occurs, release locks, abort and deliver that fault; 
IF (DS:TMP_GPR does not resolve to EPC page) 
THEN #PF(DS:TMP_GPR); FI; 
IF (EPCM(DS:TMP_GPR).VALID = 0) 
THEN #PF(DS:TMP_GPR); FI; 
IF (EPCM(DS:TMP_GPR).BLOCKED = 1) 
THEN #PF(DS:TMP_GPR); FI; 


IF ((EPCM(DS:TMP_GPR).PENDING = 1) or (EPCM(DS:TMP_GPR).MODIFIED = 1)) 
THEN #PF(DS:TMP_GPR); Fl; 
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IF ( ( EPCM(DS:TMP_GPR).ENCLAVEADDRESS # DS:TMP_GPR) or (EPCM(DS:TMP_GPR).PT # PT_REG) or 
(EPCM(DS:TMP_GPR).ENCLAVESECS # EPCM(DS:RBX).ENCLAVESECS) or 
(EPCM(DS:TMP_GPR).R = 0) or (EPCM(DS:TMP_GPR).W = 0) ) 


THEN #PF(DS:TMP_GPR),; Fl; 


IF (TMP_MODE64 = 0) 
THEN 
IF (TMP_GPR + (GPR_SIZE -1) is not in DS segment) THEN #GP(0); FI; 
Fl; 


CR_GPR_PA := Physical_Address (DS: TMP_GPR); 


TMP_TARGET := (DS:TMP_GPR).RIP; 
IF (TMP_MODE64 = 1) 
THEN 
IF (TMP_TARGET is not canonical) THEN #GP(0); FI; 
ELSE 
IF (TMP_TARGET > CS limit) THEN #GP(0); Fl; 
FI; 


(* Check proposed FS/GS segments fall within DS *) 
IF (TMP_MODE64 = 0) 
THEN 
TMP_FSBASE := (DS:RBX).OFSBASE + TMP_SECS.BASEADDR; 
TMP_FSLIMIT := (DS:RBX).OFSBASE + TMP_SECS.BASEADDR + (DS:RBX).FSLIMIT; 
TMP_GSBASE := (DS:RBX).OGSBASE + TMP_SECS.BASEADDR; 
TMP_GSLIMIT := (DS:RBX).OGSBASE + TMP_SECS.BASEADDR + (DS:RBX).GSLIMIT; 
(* if FS wrap-around, make sure DS has no holes*) 
IF (TMP_FSLIMIT < TMP_FSBASE) 
THEN 
IF (DS.limit < 4GB) THEN #GP(0); FI; 
ELSE 
IF (TMP_FSLIMIT > DS.limit) THEN #GP(0); Fl; 
FI; 
(* if GS wrap-around, make sure DS has no holes*) 
IF (TMP_GSLIMIT < TMP_GSBASE) 
THEN 
IF (DS.limit < 4GB) THEN #GP(0); FI; 
ELSE 
IF (TMP_GSLIMIT > DS.limit) THEN #GP(0); FI; 
FI; 
ELSE 
TMP_FSBASE := DS:TMP_GPR.FSBASE; 
TMP_GSBASE := DS:TMP_GPR.GSBASE; 
IF ((TMP_FSBASE is not canonical) or (TMP_GSBASE is not canonical)) 
THEN #GP(0); Fl; 
FI; 


(* Ensure the enclave is not already active and this thread is the only one using the TCS*) 
IF (DS:RBX.STATE = ACTIVE)) 
THEN #GP(0); Fl; 


TMP_IA32_U_CET := 0 
TMP_SSP := 0 
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IF (CPUID.(EAX=1 2H, ECX=1):€AX[6] = 1) 


THEN 
IF (CR4.CET = 0) 
THEN 
(* If part does not support CET or CET has not been enabled and enclave requires CET then fail *) 
IF (TMP_SECS.CET_ATTRIBUTES # 0 OR TMP_SECS.CET_LEG_BITMAP_OFFSET # 0 ) #GP(0); FI; 
Fl; 


(* If indirect branch tracking or shadow stacks enabled but CET state save area is not 16B aligned then fail ERESUME *) 
IF (TMP_SECS.CET_ATTRIBUTES.SH_STK_EN = 1 OR TMP_SECS.CET_ATTRIBUTES.ENDBR_EN = 1 ) 
THEN 
IF (DS:RBX.OCETSSA is not 16B aligned) #GP(0); FI; 
Fl; 


IF (TMP_SECS.CET_ATTRIBUTES.SH_STK_EN OR TMP_SECS.CET_ATTRIBUTES.ENDBR_EN) 
THEN 

(* Setup CET state from SECS, note tracker goes to IDLE *) 

TMP_IA32_U_CET = TMP_SECS.CET_ATTRIBUTES; 

IF (TMP_IA32_U_CET.LEG_IW_EN = 1 AND TMP_IA32_U_CET.ENDBR_EN = 1 ) 

THEN 

TMP_IA32_U_CET := TMP_IA32_U_CET + TMP_SECS.BASEADDR; 
TMP_IA32_U_CET := TMP_IA32_U_CET + TMP_SECS.CET_LEG_BITMAP_BASE; 


Fl; 


(* Compute linear address of what will become new CET state save area and cache its PA *) 
TMP_CET_SAVE_AREA = DS:RBX.OCETSSA + TMP_SECS.BASEADDR + (DS:RBX.CSSA - 1) * 16 
TMP_CET_SAVE_PAGE = TMP_CET_SAVE_AREA & ~OxFFF; 


Check the TMP_CET_SAVE_PAGE page is read/write accessible 
If fault occurs release locks, abort and deliver fault 


(* read the EPCM VALID, PENDING, MODIFIED, BLOCKED and PT fields atomically *) 
IF ((DS:TMP_CET_SAVE_PAGE Does NOT RESOLVE TO EPC PAGE) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).VALID = 0) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).PENDING = 1) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).MODIFIED = 1) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).BLOCKED = 1) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).R = 0) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).W = 0) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).ENCLAVEADDRESS # DS:TMP_CET_SAVE_PAGE) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).PT # PT_SS_REST) OR 
(EPCM(DS:TMP_CET_SAVE_PAGE).ENCLAVESECS # EPCM(DS:RBX).ENCLAVESECS)) 

THEN 

#PF(DS:TMP_CET_SAVE_PAGE); 


YS YS YS YS DS DS DS ae 


Fl; 


CR_CET_SAVE_AREA_PA := Physical address(DS:TMP_CET_SAVE_AREA) 


TMP_SSP = CR_CET_SAVE_AREA_PA.SSP 
TMP_IA32_U_CET.TRACKER = CR_CET_SAVE_AREA_PA.TRACKER; 
TMP_IA32_U_CET.SUPPRESS = CR_CET_SAVE_AREA_PA.SUPPRESS; 


IF ((TMP_MODE64 = 1 AND TMP_SSP is not canonical) OR 
(TMP_MODE64 = 0 AND (TMP_SSP & OxFFFFFFFFOOOO0000) # 0) OR 
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(TMP_SSP is not 4 byte aligned) OR 

(TMP_IA32_U_CET.TRACKER = WAIT_FOR_ENDBRANCH AND TMP_IA32_U_CET.SUPPRESS = 1) OR 
(CR_CET_SAVE_AREA_PA Reserved # 0) ) #GP(0); FI; 

FI; 


Fl; 


(* SECS.ATTRIBUTES.XFRM selects the features to be saved. *) 
(* CR_XSAVE_PAGE_n: A list of 1 or more physical address of pages that contain the XSAVE area. *) 
XRSTOR(TMP_MODE64, SECS.ATTRIBUTES.XFRM, CR_XSAVE_PAGE_n); 


IF (XRSTOR failed with #GP) 
THEN 
DS:RBX.STATE := INACTIVE; 
#GP(0); 
Fl; 


CR_ENCLAVE_MODE := 1; 
CR_ACTIVE_SECS := TMP_SECS; 
CR_ELRANGE := (TMP_SECS.BASEADDR, TMP_SECS.SIZE); 


(* Save sate for possible AEXs *) 
CR_TCS_PA := Physical_Address (DS:RBX); 
CR_TCS_LA := RBX; 

CR_TCS_LA.AEP := RCX; 


(* Save the hidden portions of FS and GS *) 
CR_SAVE_FS_selector := FS.selector; 
CR_SAVE_FS_base := FS.base; 
CR_SAVE_FS_limit := FS.limit; 
CR_SAVE_FS_access_rights := FS.access_rights; 
CR_SAVE_GS_selector := GS.selector; 
CR_SAVE_GS_base := GS.base; 
CR_SAVE_GS_limit := GS.limit; 
CR_SAVE_GS_access_rights := GS.access_rights; 


RIP := TMP_TARGET; 
Restore_GPRs from DS:TMP_GPR; 


(*Restore the RFLAGS values from SSA*) 
RFLAGS.CF := DS:TMP_GPR.RFLAGS.CF; 
RFLAGS.PF := DS:TMP_GPR.RFLAGS.PF; 
RFLAGS.AF := DS:TMP_GPR.RFLAGS.AF; 
RFLAGS.ZF := DS:TMP_GPR.RFLAGS.ZF; 
RFLAGS.SF := DS:TMP_GPR.RFLAGS.SF; 
RFLAGS.DF := DS:TMP_GPR.RFLAGS.DF; 
RFLAGS.OF := DS:TMP_GPR.RFLAGS.OF; 
RFLAGS.NT := DS:TMP_GPR.RFLAGS.NT; 
RFLAGS.AC := DS:TMP_GPR.RFLAGS.AC; 
RFLAGS.ID := DS:TMP_GPR.RFLAGS.ID; 
RFLAGS.RF := DS:TMP_GPR.RFLAGS.RF; 
RFLAGS.VM := 0; 
IF (RFLAGS.IOPL = 3) 

THEN RFLAGS.IF := DS:TMP_GPR.RFLAGS.IF; Fl; 
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IF (TCS.FLAGS.OPTIN = 0) 
THEN RFLAGS.TF := 0; FI; 


(* If XSAVE is enabled, save XCRO and replace it with SECS.ATTRIBUTES.XFRM*) 
IF (CR4.0SXSAVE = 1) 

CR_SAVE_XCRO := XCRO; 

XCRO := TMP_SECS.ATTRIBUTES.XFRM; 
FI; 


(* Pop the SSA stack*) 
(DS:RBX).CSSA := (DS:RBX).CSSA -1; 


(* Do the FS/GS swap *) 
FS.base := TMP_FSBASE; 
FS.limit := DS:RBX.FSLIMIT; 
FS.type := 00016; 

FS.W := DS.W; 

FSS := 1; 

FS.DPL := DS.DPL; 
FS.G:= 1; 

FS.B := 1; 

FS.P := 1; 

FS.AVL := DS.AVL; 

FS.L := DS.L; 

FS.unusable := 0; 
FS.selector := OBH; 


GS.base := TMP_GSBASE; 
GS.limit := DS:RBX.GSLIMIT; 
GS.type := 00016; 

GS.W := DS.W; 

GSS := 1; 

GS.DPL := DS.DPL; 
GS.G:= 1; 

GS.B:= 1; 

GS.P := 1; 

GS.AVL := DS.AVL; 

GS.L := DS.L; 
GS.unusable := 0; 
GS.selector := OBH; 


CR_DBGOPTIN := TCS.FLAGS.DBGOPTIN; 
Suppress all code breakpoints that are outside ELRANGE; 


IF (CR_DBGOPTIN = 0) 

THEN 
Suppress all code breakpoints that overlap with ELRANGE; 
CR_SAVE_TF := RFLAGS.TF; 
RFLAGS.TF := 0; 
Suppress any MTF VM exits during execution of the enclave; 
Clear all pending debug exceptions; 
Clear any pending MTF VM exit; 

ELSE 


INTEL® SGX INSTRUCTION REFERENCES 


Vol. 3D 36-135 


INTEL® SGX INSTRUCTION REFERENCES 


Clear all pending debug exceptions; 
Clear pending MTF VM exits; 


Fl; 


IF ((CPUID.(EAX=7H, ECX=0):EDX[CET_IBT] = 1) OR (CPUID.(EAX=7, ECX=0):ECX[CET_SS] = 1) 


THEN 


(* Save enclosing application CET state into save registers *) 
CR_SAVE_IA32_U_CET := IA32_U_CET 

(* Setup enclave CET state *) 

IF CPUID.(EAX=07H, ECX=00h):ECX[CET_SS] = 1 


THEN 


CR_SAVE_SSP := SSP 
SSP := TMP_SSP; 


Fl; 


IA32_U_CET := TMP_IA32_U_CET; 


Fl; 


(* Assure consistent translations *) 


Flush_linear_context; 
Clear_Monitor_FSM; 


Allow_front_end_to_begin_fetch_at_new_RIP; 


Flags Affected 


RFLAGS.TF is cleared on opt-out entry 


Protected Mode Exceptions 


#GP(0) 


#PF(error code) 


If DS:RBX is not page aligned. 

If the enclave is not initialized. 

If the thread is not in the INACTIVE state. 

If CS, DS, ES or SS bases are not all zero. 

If executed in enclave mode. 

If part or all of the FS or GS segment specified by TCS is outside the DS segment. 
If any reserved field in the TCS FLAG is set. 

If the target address is not within the CS segment. 

If CR4.OSFXSR = 0. 

If CR4.0SXSAVE = 0 and SECS.ATTRIBUTES.XFRM # 3. 

If CR4.0SXSAVE = land SECS.ATTRIBUTES.XFRM is not a subset of XCRO. 
If a page fault occurs in accessing memory. 

If DS:RBX does not point to a valid TCS. 


If one or more pages of the current SSA frame are not readable/writable, or do not resolve to 
a valid PT_REG EPC page. 


64-Bit Mode Exceptions 


#GP(0) 
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If DS:RBX is not page aligned. 

If the enclave is not initialized. 

If the thread is not in the INACTIVE state. 

If CS, DS, ES or SS bases are not all zero. 

If executed in enclave mode. 

If part or all of the FS or GS segment specified by TCS is outside the DS segment. 
If any reserved field in the TCS FLAG is set. 


#PF(error code) 
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If the target address is not canonical. 

If CR4.OSFXSR = 0. 

If CR4.0SXSAVE = 0 and SECS.ATTRIBUTES.XFRM # 3. 

If CR4.0SXSAVE = land SECS.ATTRIBUTES.XFRM is not a subset of XCRO. 
If a page fault occurs in accessing memory operands. 

If DS:RBX does not point to a valid TCS. 


If one or more pages of the current SSA frame are not readable/writable, or do not resolve to 
a valid PT_REG EPC page. 
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36.5 INTEL® SGX VIRTUALIZATION LEAF FUNCTION REFERENCE 


Leaf functions available with the ENCLV instruction mnemonic are covered in this section. In general, each instruc- 
tion leaf requires EAX to specify the leaf function index and/or additional implicit registers specifying leaf-specific 
input parameters. An instruction operand encoding table provides details of each implicit register usage and asso- 
ciated input/output semantics. 

In many cases, an input parameter specifies an effective address associated with a memory object inside or outside 
the EPC, the memory addressing semantics of these memory objects are also summarized in a separate table. 
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EDECVIRTCHILD—Decrement VIRTCHILDCNT in SECS 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = OOH IR VIV EAX[5] This leaf function decrements the SECS VIRTCHILDCNT field. 
ENCLV[EDECVIRTCHILD] 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
IR EDECVIRTCHILD (In) Return error code (Out) | Address of an enclave page (In) | Address of an SECS page (In) 
Description 


This instruction decrements the SECS VIRTCHILDCNT field. This instruction can only be executed when current 


privilege level is 0. 


The content of RCX is an effective address of an EPC page. The DS segment is used to create linear address. 


Segment override is not supported. 


EDECVIRTCHILD Memory Parameter Semantics 


EPCPAGE 


Read/Write access permitted by Non Enclave 


SECS 


Read access permitted by Enclave 


The instruction faults if any of the following: 


EDECVIRTCHILD Faulting Conditions 


A memory operand effective address is outside the DS segment 
limit (32b mode). 


A page fault occurs in accessing memory operands. 


DS segment is unusable (32b mode). 


RBX does not refer to an enclave page (REG, TCS, TRIM, SECS). 


A memory address is in a non-canonical form (64b mode). 


RCX does not refer to an SECS page. 


A memory operand is not properly aligned. 


RBX does not refer to an enclave page associated with SECS 
referenced in RCX. 


Concurrency Restrictions 


Table 36-74. Base Concurrency Restrictions of EDECVIRTCHILD 


Base Concurrency Restrictions 
Leaf Parameter i 
Access On Conflict sG7_CONFUCT VM pa 
Qualification 
EDECVIRTCHILD Target [DS:RBX] Shared SGX_EPC_PAGE_ 
CONFLICT 
SECS [DS:RCX] Concurrent 
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Table 36-75. Additional Concurrency Restrictions of EDECVIRTCHILD 


Additional Concurrency Restrictions 


vs. EACCEPT, EACCEPTCOPY, 


Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 

EDECVIRTCHILD | Target [DS:RBX] | Concurrent Concurrent Concurrent 

SECS [DS:RCX] _ | Concurrent Concurrent Concurrent 

Operation 
Temp Variables in EDECVIRTCHILD Operational Flow 
Name Type Size (bits) | Description 
TMP_SECS Physical Address 64 Physical address of the SECS of the page being modified. 
TMP_VIRTCHILDCNT Integer 64 Number of virtual child pages. 
EDECVIRTCHILD Return Value in RAX 

Error Value Description 
No Error 0 EDECVIRTCHILD Successful. 


SGX_EPC_PAGE_CONFLICT 


Failure due to concurrent operation of another SGX instruction. 


SGX_INVALID_COUNTER 


Attempt to decrement counter that is already zero. 


(* check alignment of DS:RBX *) 
IF (DS:RBX is not 4K aligned) THEN 
#GP(0); Fl; 


(* check DS:RBX is an linear address of an EPC page *) 


IF (DS:RBX does not resolve within an EPC) THEN 
#PF(DS:RBX, PFEC.SGX); Fl; 


(* check DS:RCX is an linear address of an EPC page *) 


IF (DS:RCX does not resolve within an EPC) THEN 
#PF(DS:RCX, PFEC.SGX); Fl; 


(* Check the EPCPAGE for concurrency *) 

IF (EPCPAGE is being modified) THEN 
RFLAGS.ZF = 1; 
RAX = SGX_EPC_PAGE_CONFLICT; 
goto DONE; 

FI; 


(* check that the EPC page is valid *) 
IF (EPCM(DS:RBX).VALID = 0) THEN 
#PF(DS:RBX, PFEC.SGX); Fl; 


(* check that the EPC page has the correct type and that the back pointer matches the pointer passed as the pointer to parent *) 


IF ((EPCM(DS:RBX).PAGE_TYPE = PT_REG) or 
(EPCM(DS:RBX).PAGE_TYPE = PT_TCS) or 
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(EPCM(DS:RBX).PAGE_TYPE = PT_TRIM) or 
(EPCM(DS:RBX).PAGE_TYPE = PT_SS_FIRST) or 
(EPCM(DS:RBX).PAGE_TYPE = PT_SS_REST)) 
THEN 

(* get the SECS of DS:RBX *) 
TMP_SECS := Address of SECS for (DS:RBX); 

ELSE IF (EPCM(DS:RBX).PAGE_TYPE = PT_SECS) THEN 
(* get the physical address of DS:RBX *) 
TMP_SECS := Physical_Address(DS:RBX); 

ELSE 
(* EDECVIRTCHILD called on page of incorrect type *) 
#PF(DS:RBX, PFEC.SGX); Fl; 


IF (TMP_SECS # Physical_Address(DS:RCX)) THEN 
#GP(0); Fl; 


(* Atomically decrement virtchild counter and check for underflow *) 
Locked_Decrement(SECS(TMP_SECS).VIRTCHILDCNT); 
IF (There was an underflow) THEN 
Locked_Increment(SECS(TMP_SECS).VIRTCHILDCNT); 
RFLAGS.ZF := 1; 
RAX := SGX_INVALID_COUNTER; 
goto DONE; 
FI; 


RFLAGS.ZF := 0; 
RAX := 0; 


DONE: 

(* clear flags *) 

RFLAGS.CF := 0; 
RFLAGS.PF := 0; 
RFLAGS.AF := 0; 
RFLAGS.OF := 0; 
RFLAGS.SF := 0; 


Flags Affected 
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ZF is set if EDECVIRTCHILD fails due to concurrent operation with another SGX instruction, or if there is a VIRT- 


CHILDCNT underflow. Otherwise cleared. 


Protected Mode Exceptions 


#GP(0) If a memory operand effective address is outside the DS segment limit. 


If DS segment is unusable. 


If a memory operand is not properly aligned. 


RBX does not refer to an enclave page associated with SECS referenced in RCX. 


#PF(error code) If a page fault occurs in accessing memory operands. 


If RBX does not refer to an enclave page (REG, TCS, TRIM, SECS). 


If RCX does not refer to an SECS page. 
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64-Bit Mode Exceptions 
#GP(0) If a memory address is in a non-canonical form. 

If a memory operand is not properly aligned. 

RBX does not refer to an enclave page associated with SECS referenced in RCX. 
#PF(error code) If a page fault occurs in accessing memory operands. 

If RBX does not refer to an enclave page (REG, TCS, TRIM, SECS). 

If RCX does not refer to an SECS page. 
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EINCVIRTCHILD—Increment VIRTCHILDCNT in SECS 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bitMode | Feature 
Support | Flag 
EAX = 01H IR VIV EAX[5] This leaf function increments the SECS VIRTCHILDCNT field. 
ENCLV[EINCVIRTCHILD] 
Instruction Operand Encoding 
Op/En EAX RBX RCX 
IR EINCVIRTCHILD (In) Return error code (Out) | Address of an enclave page (In) | Address of an SECS page (In) 
Description 


This instruction increments the SECS VIRTCHILDCNT field. This instruction can only be executed when the current 


privilege level is 0. 


The content of RCX is an effective address of an EPC page. The DS segment is used to create a linear address. 


Segment override is not supported. 


EINCVIRTCHILD Memory Parameter Semantics 


EPCPAGE 


Read/Write access permitted by Non Enclave 


SECS 


Read access permitted by Enclave 


The instruction faults if any of the following: 


EINCVIRTCHILD Faulting Conditions 


A memory operand effective address is outside the DS segment 
limit (32b mode). 


A page fault occurs in accessing memory operands. 


DS segment is unusable (32b mode). 


RBX does not refer to an enclave page (REG, TCS, TRIM, SECS). 


A memory address is in a non-canonical form (64b mode). 


RCX does not refer to an SECS page. 


A memory operand is not properly aligned. 


RBX does not refer to an enclave page associated with SECS 
referenced in RCX. 


Concurrency Restrictions 


Table 36-76. Base Concurrency Restrictions of EINCVIRTCHILD 


Base Concurrency Restrictions 
Leaf Parameter F 
Access On Conflict SG CONEIET Vi eat 
Qualification 
EINCVIRTCHILD Target [DS:RBX] Shared SGX_EPC_PAGE_ 
CONFLICT 
SECS [DS:RCX] Concurrent 
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Table 36-77. Additional Concurrency Restrictions of EINCVIRTCHILD 


Additional Concurrency Restrictions 
vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
EINCVIRTCHILD | Target [DS:RBX] | Concurrent Concurrent Concurrent 
SECS [DS:RCX] _ | Concurrent Concurrent Concurrent 
Operation 

Temp Variables in EINCVIRTCHILD Operational Flow 
Name Type Size (bits) | Description 
TMP_SECS Physical Address 64 Physical address of the SECS of the page being modified. 

EINCVIRTCHILD Return Value in RAX 

Error Value Description 
No Error 0 EINCVIRTCHILD Successful. 
SGX_EPC_PAGE_CONFLICT Failure due to concurrent operation of another SGX instruction. 


(* check alignment of DS:RBX *) 
IF (DS:RBX is not 4K aligned) THEN 


#GP(0); Fl; 


(* check DS:RBX is an linear address of an EPC page *) 


IF (DS:RBX does not resolve within an EPC) THEN 
#PF(DS:RBX, PFEC.SGX); FI; 


(* check DS:RCX is an linear address of an EPC page *) 


IF (DS:RCX does not resolve within an EPC) THEN 
#PF(DS:RCX, PFEC.SGX); Fl; 


(* Check the EPCPAGE for concurrency *) 
IF (EPCPAGE is being modified) THEN 


RFLAGS.ZF = 1; 


RAX = SGX_EPC_PAGE_CONFLICT; 


goto DONE; 
Fl; 


(* check that the EPC page is valid *) 


IF (EPCM(DS:RBX).VALID = 0) THEN 
#PF(DS:RBX, PFEC.SGX); Fl; 


(* check that the EPC page has the correct type and that the back pointer matches the pointer passed as the pointer to parent *) 

IF ((EPCM(DS:RBX).PAGE_TYPE = PT_REG) or 
(EPCM(DS:RBX).PAGE_TYPE = PT_TCS) or 
(EPCM(DS:RBX).PAGE_TYPE = PT_TRIM) or 


( ). 
(EPCM(DS:RBX). 
( ). 


PAGE_TYPE = PT_SS_FIRST) or 


(EPCM(DS:RBX).PAGE_TYPE = PT_SS_REST)) 
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THEN 
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(* get the SECS of DS:RBX *) 
TMP_SECS := Address of SECS for (DS:RBX); 
ELSE IF (EPCM(DS:RBX).PAGE_TYPE = PT_SECS) THEN 


(* get the physical address of DS:RBX * 
TMP_SECS := Physical_Address(DS:RBX 


ELSE 


Cow 


fi 


(* EINCVIRTCHILD called on page of incorrect type *) 
#PF(DS:RBX, PFEC.SGX); Fl; 


IF (TMP_SECS # Physical_Address(DS:RCX)) THEN 


#GP(0); Fl; 


(* Atomically increment virtchild counter *) 
Locked_Increment(SECS(TMP_SECS).VIRTCHILDCNT); 


RFLAGS.ZF := 0; 
RAX := 0; 


DONE: 

(* clear flags *) 

RFLAGS.CF := 0; 
RFLAGS.PF := 0; 
RFLAGS.AF := 0; 
RFLAGS.OF := 0; 
RFLAGS.SF := 0; 


Flags Affected 


ZF is set if EINCVIRTCHILD fails due to concurrent operation with another SGX instruction; otherwise cleared. 


Protected Mode Exceptions 


#GP(0) 


#PF(error code) 


If a memory operand effective address is outside the DS segment limit. 
If DS segment is unusable. 
If a memory operand is not properly aligned. 


RBX does not refer to an enclave page associated with SECS referenced in RCX. 


If a page fault occurs in accessing memory operands. 
If RBX does not refer to an enclave page (REG, TCS, TRIM, SECS). 
If RCX does not refer to an SECS page. 


64-Bit Mode Exceptions 


#GP(0) 


#PF(error code) 


If a memory address is in a non-canonical form. 
If a memory operand is not properly aligned. 


RBX does not refer to an enclave page associated with SECS referenced in RCX. 


If a page fault occurs in accessing memory operands. 
If RBX does not refer to an enclave page (REG, TCS, TRIM, SECS). 
If RCX does not refer to an SECS page. 
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ESETCONTEXT—Set the ENCLAVECONTEXT Field in SECS 


Opcode/ Op/En | 64/32 CPUID Description 
Instruction bit Mode | Feature 
Support | Flag 
EAX = 02H IR VN EAX[5] This leaf function sets the ENCLAVECONTEXT field in SECS. 
ENCLV[ESETCONTEXT] 


Instruction Operand Encoding 


Op/en EAX ROX RDX 
IR ESETCONTEXT (In) | Return error code (Out) | Ad¢ress of saree as EPC page | Context Value (In, EA) 


Description 


The ESETCONTEXT leaf overwrites the ENCLAVECONTEXT field in the SECS. ECREATE and ELD of an SECS set the 
ENCLAVECONTEXT field in the SECS to the address of the SECS (for access later in ERDINFO). The ESETCONTEXT 
instruction allows a VMM to overwrite the default context value if necessary, for example, if the VMM is emulating 
ECREATE or ELD on behalf of the guest. 


The content of RCX is an effective address of the SECS page to be updated, RDX contains the address pointing to 
the value to be stored in the SECS. The DS segment is used to create linear address. Segment override is not 
supported. 


The instruction fails if: 
¢ The operand is not properly aligned. 
* RCX does not refer to an SECS page. 


ESETCONTEXT Memory Parameter Semantics 


EPCPAGE CONTEXT 


Read access permitted by Enclave Read/Write access permitted by Non Enclave 


The instruction faults if any of the following: 


ESETCONTEXT Faulting Conditions 


A memory operand effective address is outside the DS segment | A memory operand is not properly aligned. 
limit (32b mode). 


DS segment is unusable (32b mode). A page fault occurs in accessing memory operands. 


A memory address is in a non-canonical form (64b mode). 


Concurrency Restrictions 


Table 36-78. Base Concurrency Restrictions of ESETCONTEXT 


Base Concurrency Restrictions 
Leaf Parameter : SGX_CONFLICT VM Exit 
Access On Conflict oe 
Qualification 
ESETCONTEXT SECS [DS:RCX] Shared SGX_EPC_PAGE_ 
CONFLICT 
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Table 36-79. Additional Concurrency Restrictions of ESETCONTEXT 


Additional Concurrency Restrictions 


vs. EACCEPT, EACCEPTCOPY, 
Leaf Parameter EMODPE, EMODPR, EMODT vs. EADD, EEXTEND, EINIT vs. ETRACK, ETRACKC 
Access On Conflict Access On Conflict Access On Conflict 
ESETCONTEXT | SECS [DS:RCX] | Concurrent Concurrent Concurrent 
Operation 

Temp Variables in ESETCONTEXT Operational Flow 
Name Type Size (bits) | Description 
TMP_SECS Physical Address 64 Physical address of the SECS of the page being modified. 
TMP_CONTEXT CONTEXT 64 Data Value of CONTEXT. 

ESETCONTEXT Return Value in RAX 

Error Value Description 
No Error 0 ESETCONTEXT Successful. 
SGX_EPC_PAGE_CONFLICT Failure due to concurrent operation of another SGX instruction. 


(* check alignment of the EPCPAGE (RCX) *) 
IF (DS:RCX is not 4KByte Aligned) THEN 
#GP(0); Fl; 


(* check that EPCPAGE (DS:RCX) is the address of an EPC page *) 


IF (DS:RCX does not resolve within an EPC)THEN 
#PF(DS:RCX, PFEC.SGX),; Fl; 


(* check alignment of the CONTEXT field (RDX) *) 


IF (DS:RDX is not 8Byte Aligned) THEN 
#GP(0); FI; 


(* Load CONTEXT into local variable *) 
TMP_CONTEXT := DS:RDX 


(* Check the EPC page for concurrency *) 
IF (EPC page is being modified) THEN 
RFLAGS.ZF := 1; 
RFLAGS.CF := 0; 
RAX := SGX_EPC_PAGE_CONFLICT; 
goto DONE; 
Fl; 


(* check page validity *) 

IF (EPCM(DS:RCX).VALID = 0) THEN 
#PF(DS:RCX, PFEC.SGX); 

Fl; 


(* check EPC page is an SECS page *) 
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IF (EPCM(DS:RCX).PT is not PT_SECS) THEN 
#PF(DS:RCX, PFEC.SGX); 
Fl; 


(* load the context value into SECS(DS:RCX).ENCLAVECONTEXT *) 
SECS(DS:RCX).ENCLAVECONTEXT := TMP_CONTEXT; 


RAX := 0; 
RFLAGS.ZF := 0; 


DONE: 
(* clear flags *) 
RFLAGS.CF,PF,AF,OF,SF := 0; 


Flags Affected 
ZF is set if ESETCONTEXT fails due to concurrent operation with another SGX instruction; otherwise cleared. 
CF, PF, AF, OF and SF are cleared. 


Protected Mode Exceptions 

#GP(0) If a memory operand effective address is outside the DS segment limit. 
If DS segment is unusable. 
If a memory operand is not properly aligned. 

#PF(error code) If a page fault occurs in accessing memory operands. 


64-Bit Mode Exceptions 
#GP(0) If a memory address is in a non-canonical form. 

If a memory operand is not properly aligned. 
#PF(error code) If a page fault occurs in accessing memory operands. 
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CHAPTER 37 
INTEL® SGX INTERACTIONS WITH IA32 AND INTEL® 64 ARCHITECTURE 


Intel® SGX provides Intel® Architecture with a collection of enclave instructions for creating protected execution 
environments on processors supporting IA32 and Intel® 64 architectures. These Intel SGX instructions are 
designed to work with legacy software and the various IA32 and Intel 64 modes of operation. 


37.1 INTEL® SGX AVAILABILITY IN VARIOUS PROCESSOR MODES 


The Intel SGX extensions (see Table 32-1) are available only when the processor is executing in protected mode of 
operation. Additionally, the extensions are not available in System Management Mode (SMM) of operation or in 
Virtual 8086 (VM86) mode of operation. Finally, all leaf functions of ENCLU and ENCLS require CRO.PG enabled. 
The exact details of exceptions resulting from illegal modes and their priority are listed in the reference pages of 
ENCLS and ENCLU. 


37.2 1IA32_FEATURE_CONTROL 


IA32_FEATURE_CONTROL MSR provides two new bits related to two aspects of Intel SGX: using the instruction 
extensions and launch control configuration. 


37.2.1 Availability of Intel SGX 


IA32_FEATURE_CONTROL[bit 18] allows BIOS to control the availability of Intel SGX extensions. For Intel SGX 
extensions to be available on a logical processor, bit 18 in the IA32_FEATURE_CONTROL MSR on that logical 
processor must be set, and IA32_ FEATURE_CONTROL MSR on that logical processor must be locked (bit O must be 
set). See Section 32.7.1 for additional details. OS is expected to examine the value of bit 18 prior to enabling Intel 
SGX on the thread, as the settings of bit 18 is not reflected by CPUID. 


37.2.2 — Intel SGX Launch Control Configuration 


The IA32_SGXLEPUBKEYHASHn MSRs used to configure authorized launch enclaves' MRSIGNER digest value. They 
are present on logical processors that support the collection of SGX1 leaf functions (i.e. CPUID.(EAX=12H, 
ECX=00H):EAX[0] = 1) and that CPUID.(EAX=07H, ECX=00H):ECX[30] = 1. IA32_FEATURE_CONTROL|[bit 17] 
allows to BIOS to enable write access to these MSRs. If IA32_FEATURE_CONTROL.LE_WR (bit 17) is set to 1 and 
IA32_FEATURE_CONTROL is locked on that logical processor, IA32_SGXLEPUBKEYHASH MSRs on that logical 
processor are writeable. If this bit 17 is not set or IA32_FEATURE_CONTROL is not locked, 
IA32_SGXLEPUBKEYHASH MSRs are read only. See Section 34.1.4 for additional details. 


37.3. INTERACTIONS WITH SEGMENTATION 


37.3.1 Scope of Interaction 


Intel SGX extensions are available only when the processor is executing in a protected mode operation (see Section 
37.1 for Intel SGX availability in various processor modes). Enclaves abide by all the segmentation policies set up 
by the OS, but they can be more restrictive than the OS. 


Intel SGX interacts with segmentation at two levels: 
¢ The Intel SGX instruction (see the enclave instruction in Table 32-1). 
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¢ While executing inside an enclave (legacy instructions and enclave instructions permitted inside an enclave). 


37.3.2 Interactions of Intel® SGX Instructions with Segment, Operand, and Addressing 
Prefixes 


All the memory operands used by the Intel SGX instructions are interpreted as offsets within the data segment 
(DS). The segment-override prefix on Intel SGX instructions is ignored. 


Operand size is fixed for each enclave instruction. The operand-size prefix is reserved, and results in a #UD excep- 
tion if used. 


All address sizes are determined by the operating mode of the processor. The address-size prefix is ignored. This 
implies that while operating in 64-bit mode of operation, the address size is always 64 bits, and while operating in 
32-bit mode of operation, the address size is always 32 bits. Additionally, when operating in 16-bit addressing, 
memory operands used by enclave instructions use 32 bit addressing; the value of CS.D is ignored. 


37.3.3. Interaction of Intel® SGX Instructions with Segmentation 


All leaf functions of ENCLU and ENCLS instructions require that the DS segment be usable, and be an expand-up 
segment. Failing this check results in generation of a #GP(0) exception. 


The Intel SGX leaf functions used for entering the enclave (ENCLU[EENTER] and ENCLU[ERESUME]) operate as 
follows: 


e All usable segment registers except for FS and GS have a zero base. 
¢ The contents of the FS/GS segment registers (including the hidden portion) is saved in the processor. 
° New FS and GS values compatible with enclave security are loaded from the TCS 


¢ The linear ranges and access rights available under the newly-loaded FS and GS must abide to OS policies by 
ensuring they are subsets of the linear-address range and access rights available for the DS segment. 

¢ The CS segment mode (64-bit, compatible, or 32 bit modes) must be consistent with the segment mode for 
which the enclave was created, as indicated by the SECS.ATTRIBUTES.MODE64 bit, and that the CPL of the 
logical processor is 3 

An exit from the enclave either via ENCLU[EEXIT] or via an AEX restores the saved values of FS/GS segment regis- 

ters. 


37.3.4 — Interactions of Enclave Execution with Segmentation 


During the course of execution, enclave code abides by all segmentation policies as dictated by IA32 and Intel 64 
Architectures, and generates appropriate exceptions on violations. 


Additionally, any attempt by software executing inside an enclave to modify the processor's segmentation state 
(e.g. via MOV seg register, POP seg register, LDS, far jump, etc; excluding WRFSBASE/WRGSBASE) results in the 
generation of a #UD. See Section 34.6.1 for more information. 


Upon enclave entry via the EENTER leaf function, FS is loaded from the (TCS.OFSBASE + SECS.BASEADDR) and 
TCS.FSLIMIT fields and GS is loaded from the (TCS.OGSBASE + SECS.BASEADDR) and TCS.GSLIMIT fields. 


Execution of WRFSBASE and WRGSBASE from inside a 64-bit enclave is allowed. The processor will save the new 
values into the current SSA frame on an asynchronous exit (AEX) and restore them back on enclave entry via 
ENCLU[ERESUME] instruction. 


37.4 INTERACTIONS WITH PAGING 


Intel SGX instructions are available only when the processor is executing in a protected mode of operation. Addi- 
tionally, all Intel SGX leaf functions except for EDBGRD and EDBGWR are available only if paging is enabled. Any 
attempt to execute these leaf functions with paging disabled results in an invalid-opcode exception (#UD). As with 
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segmentation, enclaves abide by all the paging policies set up by the OS, but they can be more restrictive than the 
Os. 


All the memory operands passed into Intel SGX instructions are interpreted as offsets within the DS segment, and 
the linear addresses generated by combining these offsets with DS segment register are subject to paging-based 
access control if paging is enabled at the time of the execution of the leaf function. 


Since the ENCLU[EENTER] and ENCLU[ERESUME] can only be executed when paging is enabled, and since paging 
cannot be disabled by software running inside an enclave (recall that enclaves always run with CPL = 3), enclave 
execution is always subject to paging-based access control. The Intel SGX access control itself is implemented as 
an extension to the existing paging modes. See Section 33.5 for details. 


Execution of Intel SGX instructions may set accessed and dirty flags on accesses to EPC pages that do not fault 
even if the instruction later causes a fault for some other reason. 


37.5 INTERACTIONS WITH VMX 


Intel SGX functionality (including SGX1 and SGX2) can be made available to software running in either VMX root 
operation or VMX non-root operation, as long as the processor is using a legal mode of operation (see Section 
37.1). 


A VMM has the flexibility to configure a VMCS to permit a guest to use any subset of the ENCLS leaf functions. Avail- 
ability of the ENCLU leaf functions in VMX non-root operation has the same requirement as ENCLU leaf functions 
outside of a virtualized environment. 


Details of the VMCS control to allow VMM to configure support of Intel SGX in VMX non-root operation is described 
in Section 37.5.1 


37.5.1 | VMM Controls to Configure Guest Support of Intel® SGX 


Intel SGX capabilities are primarily exposed to the software via the CPUID instruction. VMMs can virtualize CPUID 
instruction to expose/hide this capability to/from guests. 


Some of Intel SGX resources are exposed/controlled via model-specific registers (see Section 32.7). VMMs can 
virtualize these MSRs for the guests using the MSR bitmaps referenced by pointers in the VMCS. 


The VMM can partition the Enclave Page Cache, and assign various partitions to (a subset of) its guests via the 
usual memory-virtualization techniques such as paging or the extended page table mechanism (EPT). 


The VMM can set the “enable ENCLS exiting” VM-execution controls to cause a VM exit when the ENCLS instruction 
is executed in VMX non-root operation. If the “enable ENCLS exiting” control is 0, all of the ENCLS leaf functions are 
permitted in VMX non-root operation. If the “enable ENCLS exiting” control is 1, execution of ENCLS leaf functions 
in VMX non-root operation is governed by consulting the bits in a new 64-bit VM-execution control field called the 
ENCLS-exiting bitmap (Each bit in the bitmap corresponds to an ENCLS leaf function with an EAX value that is iden- 
tical to the bit’s position). When bits in the “ENCLS-exiting bitmap” are set, attempts to execute the corresponding 
ENCLS leaf functions in VMX non-root operation causes VM exits. The checking for these VM exits occurs immedi- 
ately after checking that CPL = 0. 


37.5.2 Interactions with the Extended Page Table Mechanism (EPT) 
Intel SGX instructions are fully compatible with the extended page-table mechanism (EPT; see Section 27.2). 


All the memory operands passed into Intel SGX instructions are interpreted as offsets within the DS segment, and 
the linear addresses generated by combining these offsets with DS segment register are subject to paging and EPT. 
As with paging, enclaves abide by all the policies set up by the VMM. 


The Intel SGX access control itself is implemented as an extension to paging and EPT, and may be more restrictive. 
See Section 37.4 for details of this extension. 


An execution of an Intel SGX instruction may set accessed and dirty flags for EPT (when enabled; see Section 
27.2.5) on accesses to EPC pages that do not fault or cause VM exits even if the instruction later causes a fault or 
VM exit for some other reason. 
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37.5.3 Interactions with APIC Virtualization 


This section applies to Intel SGX in VMX non-root operation when the “virtualize APIC accesses” VM-execution 
control is 1. 


A memory access by an enclave instruction that implicitly uses a cached physical address is never checked for 
overlap with the APIC-access page. Such accesses never cause APIC-access VM exits and are never redirected to 
the virtual-APIC page. Implicit memory accesses can only be made to the SECS, the TCS, or the SSA of an enclave 
(see Section 33.5.3.2). 


An explicit Enclave Access (a linear memory access which is either from within an enclave into its ELRANGE, or an 
access by an Intel SGX instruction that is expected to be in the EPC) that overlaps with the APIC-access page 
causes a #PF exception (APIC page is expected to be outside of EPC). 

Non-Enclave accesses made either by an Intel SGX instruction or by a logical processor inside an enclave to an 
address that without SGX would have caused redirection to the virtual-APIC page instead cause an APIC-access 
VM exit. 


Other than implicit accesses made by Intel SGX instructions, guest-physical and physical accesses are not consid- 
ered “enclave accesses”; consequently, such accesses result in undefined behavior if these accesses eventually 
reach EPC. This applies to any non-enclave physical accesses. 


While a logical processor is executing inside an enclave, an attempt to execute an instruction outside of ELRANGE 
results in a #GP(0), even if the linear address would translate to a physical address that overlaps the APIC-access 


page. 


37.5.4 Interactions with VT and SGX concurrency 


In some cases, a VMM is required to handle conflicts between its own operation and a guest operation on EPC pages 
that are present in both guest and VMM address space. These conflict would otherwise cause the guest to experi- 
ence an unexpected behavior (vs. running directly on the h/w). These conflict cases are: 


¢ ETRACK/ETRACKC failure due to Entry Epoch Object Lock conflict or reference tracking check failure. 
¢ EPC Page Resource conflict. 


A new exit reason is defined for all those cases: SGX_CONFLICT (value 71). The VMCS exit qualification field details 
the specific case as follows: 


Table 37-1. SGX Conflict Exit Qualification 


Bits Size (bits) Name Description 


15:0 16 Code Exit qualification code. The following values are defined: 
0: TRACKING_RESOURCE_CONFLICT 

1: TRACKING_REFERENCE_CONFLICT 

2: EPC_PAGE_CONFLICT_EXCEPTION 

3: EPC_PAGE_CONFLICT_ERROR 

Other: Reserved 


31:16 16 Error Error code. Applicable only if the exit qualification code is EPC_PAGE_CONFLICT_ERROR; 
contains the error code that would be returned in RAX if the instruction was executed on 
bare metal platform or if the ENABLE_EPC_VIRTUALIZATION_EXTENSIONS bit in the 
secondary processor control field is not set. In other cases this field is reserved as O. 


63:32 |32 Reserved Always 0. 


This SGX_CONFLICT exiting behavior is controlled by a VM execution control called 
ENABLE_EPC_VIRTUALIZATION_EXTENSIONS (bit 29 of the secondary processor control field). 


Details for various SGX_CONFLICT VMEXIT cases are provided in the following sections. 
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37.5.5 Virtual Child Tracking 


SGX oversubscription support adds the ability to associate virtual children with each enclave using the 
ENCLV[EINCVIRTCHILD] and ENCLV[EDECVIRTCHILD] instructions. The VMM enables checking of the virtual child 
count by EREMOVE and EWB in guests with a new VM execution control called 
ENABLE_EPC_VIRTUALIZATION_EXTENSIONS. 


When in VMX non-root operation and the ENABLE_EPC_VIRTUALIZATION_EXTENSIONS control enabled, the 
following instructions change their behavior: 


¢ EWB and EREMOVE return the SGX_CHILD_PRESENT error code if any virtual or physical children are 
associated with the enclave. 


¢ ERDINFO set STATUS.CHILDPRESENT if any virtual or physical children are associated with the enclave. 


37.5.6 Handling EPCM Entry Lock Conflicts 


When performing paging within a VMM, it is possible for a contention on the EPC page to happen in the following 
case: 


¢ The VMM performs an ELDB/ELDU/ELDBC/ELDUC of an enclave page, and the guest attempts to perform some 
SGX instruction (e.g., EREMOVE) where the same SECS parent page is required. 


A similar conflict may occur if the VMM uses EINCVIRTCHILD or EDECVIRTCHILD pointing to an SECS page. In all 
other cases where a SGX instruction executed by the VMM the applicable EPC page should not be mapped to the 
guest, thus no resource conflict occurs. 


This conflicting situation can cause the guest's instruction to fail and cause guest instability. To help the VMM 
manage such conflicts, the SGX VMM paging extensions introduce a new VM-Exit that will be triggered whenever 
the guest encounters a resource conflict. 


The exit reason is SGX_CONFLICT. The exit qualification field is used to distinguish the two kinds of resource 
conflicts: 


¢ = Avalue of EPCM_RESOURCE_CONFLICT_EXCEPTION (2) in the exit qualification code field indicates that a 
resource conflict occurred that would result in a #GP. In that case, the exit qualification error field is set to zero. 


¢ Avalue of EPC_PAGE_CONFLICT_ERROR (3) in the exit qualification code field indicates that a resource conflict 
occurred that would result in an error code being return in RAX. In that case, the exit qualification error field is 
set to SGX_EPC_PAGE_CONFLICT. 


The Guest Linear Address and Guest Physical Address fields are set to the guest linear and guest physical 
addresses respectively of the EPC page on which the conflict occurred. The VMM may determine which instruction 
induced the exit by reading RAX. The exit also populates the VM-exit instruction length field. 


The VMM can determine whether the conflict may be due to its own operation, e.g., by setting a per-enclave busy 
indicator before executing ELD*, and clearing it afterwards. In that case, the VMM can handle an SGX Conflict 
(EPCM_PAGE_CONFLICT_*) exit by resuming guest execution at the same instruction, allowing the guest to re- 
execute the instruction. The VMM may also take steps to throttle its own paging thread to reduce contention with 
the guest. 


If the VMM determines that the conflict is not due to its own operation, it may inject a #GP (in case of 
EPC_PAGE_CONFLICT_EXCEPTION), or emulate an error code as the guest instruction would return (in case of 
EPC_PAGE_CONFLICT_ERROR) by setting ZF and copying the error value provided in the exit qualification to guest 
RAX. 


To gracefully handle resource contention on the VMM side, the VMM should use the new ELDBC and ELDUC instruc- 
tions. These are similar to ELDB and ELDU respectively, except that on EPC resource contention they return an 
SGX_EPC_PAGE_CONFLICT error instead of issuing a #GP. In case of an error, the VMM can retry the instruction, 
possibly throttling the guest to assure progress. 


When using EDECVIRTCHILD and EINCVIRTCHILD, the VMM should preferably point to the enclave child page, not 
to the SECS page, avoiding resource conflict on the SECS. If the VMM chooses to point to the SECS page, it should 
handle conflicts in the same way as handling the ELD* case. 
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37.5.7. Context Tracking 


The ENCLAVECONTEXT field in the SECS is available for use by the VMM to track context information associated 
with that enclave, such as the GPA of the SECS in the context of the appropriate guest. This field is initialized by the 
successful execution of ECREATE and ELD of an SECS page. The value stored in the ENCLAVECONTEXT field will be 
the translation of the target page address produced by paging (GPA in VMMs that have EPTs turned on). VMMs may 
override this default value by calling the ENCLV[ESETCONTEXT] instruction, which allows the VMM to store an arbi- 
trary 64-bit value in the ENCLAVECONTEXT field. The VMM may later access the ENCLAVECONTEXT field by calling 
ENCLS[ERDINFO] on any member page of the enclave, including the SECS. 


For nested virtualization cases, the lowest level VMM can make SGX oversubscription instructions higher level 
guest VMMs. In that case the lower level VMM can simply inject #GP to higher level VMMs when attempting to 
execute these instructions. 


However, if VMMs expose SGX oversubscription instructions to higher level VMMs, then VMMs have to use 
ENCLV[ESETCONTEXT] instruction to properly manage the ENCLAVECONTEXT field of SECS during paging opera- 
tions. That may involve emulating ECREATE, ELD, ESETCONTEXT and ERDINFO instructions apart from managing 
ENCLAVECONTEXT values. 


37.6 INTEL® SGX INTERACTIONS WITH ARCHITECTURALLY-VISIBLE EVENTS 


All architecturally visible events (exceptions, interrupts, SMI, NMI, INIT, VM exit) can be detected while inside an 
enclave and will cause an asynchronous enclave exit if they are not blocked. Additionally, INT3, and the 
SignalTXTMsg[SENTER] (i.e. GETSEC[SENTER]’s rendezvous event message) events also cause asynchronous 
enclave exits. Note that SignalTXTMsg[SEXIT] (i.e. GETSEC[SEXIT]’s teardown message) does not cause an AEX. 


On an AEX, information about the event causing the AEX is stored in the SSA (see Section 35.4 for details of AEX). 
The information stored in the SSA only describes the first event that triggered the AEX. If parsing/delivery of the 

first event results in detection of further events (e.g. VM exit, double fault, etc.), then the event information in the 
SSA is not updated to reflect these subsequently detected events. 


37.7. INTERACTIONS WITH THE PROCESSOR EXTENDED STATE AND 
MISCELLANEOUS STATE 


37.7.1 Requirements and Architecture Overview 


Processor extended states are the ISA features that are enabled by the settings of CR4.OSXSAVE and the XCRO 
register. Processor extended states are normally saved/restored by software via XSAVE/XRSTOR instructions. 
Details of discovery of processor extended states and management of these states are described in CHAPTER 13 of 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A. 

Additionally, the following requirements apply to Intel SGX: 

® Onan AEX, the Intel SGX architecture must protect the processor extended state and miscellaneous state by 


saving them in the enclave’s state-save area (SSA), and clear the secrets from the processor extended state 
that is used by an enclave. 


¢ Intel SGX architecture must verify that the SSA frame size is large enough to contain all the processor extended 
states and miscellaneous state used by the enclave. 


¢ Intel SGX architecture must ensure that enclaves can only use processor extended state that is enabled by 
system software in XCRO. 


® Enclave software should be able to discover only those processor extended state and miscellaneous state for 
which such protection is enabled. 


° The processor extended states that are enabled inside the enclave must be approved by the enclave developer: 


— Certain processor extended state (e.g., Memory Protection Extensions, see Chapter 17, “Intel® MPX” of 
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1) modify the behavior of the 
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legacy ISA software. If such features are enabled for enclaves that do not understand those features, then 
such a configuration could lead to a compromise of the enclave's security. 


¢ The processor extended states that are enabled inside the enclave must form an integral part of the enclave's 
identity. This requirement has two implications: 


— Service providers may decide to assign different trust level to the same enclave depending on the ISA 
features the enclave is using. 


To meet these requirements, the Intel SGX architecture defines a sub-field called X-Feature Request Mask (XFRM) 
in the ATTRIBUTES field of the SECS. On enclave creation (ENCLS[ECREATE] leaf function), the required SSA frame 
size is calculated by the processor from the list of enabled extended and miscellaneous states and verified against 
the actual SSA frame size defined by SECS.SSAFRAMESIZE. 


On enclave entry, after verifying that XFRM is only enabling features that are already enabled in XCRO, the value in 
the XCRO is saved internally by the processor, and is replaced by the XFRM. On enclave exit, the original value of 
XCRO is restored. Consequently, while inside the enclave, the processor extended states enabled in XFRM are in 
enabled state, and those that are disabled in XFRM are in disabled state. 


The entire ATTRIBUTES field, including the XFRM subfield is integral part of enclave's identity (i.e., its value is 
included in reports generated by ENCLU[EREPORT], and select bits from this field can be included in key-derivation 
for keys obtained via the ENCLU[EGETKEY] leaf function). 


Enclave developers can create their enclave to work with certain features and fallback to another code path in case 
those features aren't available (e.g. optimize for AVX and fallback to SSE). For this purpose Intel SGX provides the 
following fields in SIGSTRUCT: ATTRIBUTES, ATTRIBUTESMASK, MISCSELECT, and MISCMASK. EINIT ensures that 
the final SECS.ATTRIBUTES and SECS.MISCSELECT comply with the enclave developer's requirements as follows: 


SIGSTRUCT.ATTRIBUTES & SIGSTRUCT.ATTRIBUTEMASK = SECS.ATTRIBUTES & SIGSTRUCT.ATTRIBUTEMASK 
SIGSTRUCT.MISCSELECT & SIGSTRUCT.MISCMASK = SECS.MISCSELECT & SIGSTRUCT.MISCMASK. 


On an asynchronous enclave exit, the processor extended states enabled by XFRM are saved in the current SSA 
frame, and overwritten by synthetic state (see Section 35.3 for the definition of the synthetic state). When the 
interrupted enclave is resumed via the ENCLU[ERESUME] leaf function, the saved state for processor extended 
states enabled by XFRM is restored. 


37.7.2 Relevant Fields in Various Data Structures 


37.7.2.1 SECS.ATTRIBUTES.XFRM 


The ATTRIBUTES field of the SECS data structure (see Section 33.7) contains a sub-field called XSAVE-Feature 
Request Mask (XFRM). Software populates this field at the time of enclave creation according to the features that 
are enabled by the operating system and approved by the enclave developer. 


Intel SGX architecture guarantees that during enclave execution, the processor extended state configuration of the 
processor is identical to what is required by the XFRM sub-field. All the processor extended states enabled in XFRM 
are saved on AEX from the enclave and restored on ERESUME. 


The XFRM sub-field has the same layout as XCRO, and has consistency requirements that are similar to those for 
XCRO. Specifically, the consistency requirements on XFRM values depend on the processor implementation and the 
set of features enabled in CR4. 


Legal values for SECS.ATTRIBUTES.XFRM conform to these requirements: 
¢ XFRM[1:0] must be set to 0x3. 


¢ If the processor does not support XSAVE, or if the system software has not enabled XSAVE, then XFRM[63:2] 
must be zero. 


¢ If the processor does support XSAVE, XFRM must contain a value that would be legal if loaded into XCRO. 


The various consistency requirements are enforced at different times in the enclave's life cycle, and the exact 
enforcement mechanisms are elaborated in Section 37.7.3 through Section 37.7.6. 


On processors not supporting XSAVE, software should initialize XFRM to 0x3. On processors supporting XSAVE, 
software should initialize XFRM to be a subset of XCRO that would be present at the time of enclave execution. 
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Because bits 0 and 1 of XFRM must always be set, the use of Intel SGX requires that SSE be enabled (CR4.OSFXSR 
= 1). 


37.7.2.2 SECS.SSAFRAMESIZE 


The SSAFRAMESIZE field in the SECS data structure specifies the number of pages which software allocated! for 
each SSA frame, including both the GPRSGX area, MISC area, the XSAVE area (x87 and XMM states are stored in 
the latter area), and optionally padding between the MISC and XSAVE area. The GPRSGX area must hold all the 
general-purpose registers and additional Intel SGX specific information. The MISC area must hold the Miscella- 
neous state as specified by SECS.MISCSELECT, the XSAVE area holds the set of processor extended states specified 
by SECS.ATTRIBUTES. XFRM (see Section 33.9 for the layout of SSA and Section 37.7.3 for ECREATE's consistency 
checks). The SSA is always in non-compacted format. 


If the processor does not support XSAVE, the XSAVE area will always be 576 bytes; a copy of XFRM (which will be 
set to 0x3) is saved at offset 512 on an AEX. 


If the processor does support XSAVE, the length of the XSAVE area depends on SECS.ATTRIBUTES.XFRM. The 
length would be equal to what CPUID.(EAX=0DH, ECX= 0):EBX would return if XCRO were set to XFRM. The 
following pseudo code illustrates how software can calculate this length using XFRM as the input parameter without 
modifying XCRO: 
offset = 576; 
size_last_x = 0; 
For x=2 to 63 
IF (XFRM[x] != 0) Then 
tmp_offset = CPUID.(EAX=0DH, ECX= x):EBX[31:0]; 
IF (tmp_offset >= offset + size_last_x) Then 
offset = tmp_offset; 
size_last_x = CPUID.(EAX=0DH, ECX= x):EAX[31:0]; 
FI; 
FI; 
EndFor 
return (offset + size_last_x); (* compute_xsave_size(XFRM), see “ECREATE—Create an SECS page in the Enclave 
Page Cache”*) 


Where the non-zero bits in XFRM are a subset of non-zero bit fields in XCRO. 


The size of the MISC region depends on the setting of SECS.MISCSELECT and can be calculated using the layout 
information described in Section 33.9.2 


37.7.2.3 XSAVE Area in SSA 
The XSAVE area of an SSA frame begins at offset 0 of the frame. 


37.7.2.4 MISC Area in SSA 
The MISC area of an SSA frame is positioned immediately before the GPRSGX region. 


37.7.2.5 SIGSTRUCT Fields 


Intel SGX provides the flexibility for an enclave developer to choose the enclave's code path according to the 
features that are enabled on the platform (e.g. optimize for AVX and fallback to SSE). See Section 37.7.1 for 
details. 


1. It is the responsibility of the enclave to actually allocate this memory. 
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SIGSTRUCT includes the following fields: 
SIGSTRUCT.ATTRIBUTES, SIGSTRUCT.ATTRIBUTEMASK, SIGSTRUCT.MISCSELECT, SIGSTRUCT.MISCMASK. 


37.7.2.6 REPORT.ATTRIBUTES.XFRM and REPORT.MISCSELECT 


The processor extended states and miscellaneous states that are enabled inside the enclave form an integral part 
of the enclave's identity and are therefore included in the enclave's report, as provided by the ENCLU[EREPORT] 
leaf function. The REPORT structure includes the enclave's XFRM and MISCSELECT configurations. 


37.7.2.7. KEYREQUEST 


An enclave developer can specify which bits out of XFRM and MISCSELECT ENCLU[EGETKEY] should include in the 
derivation of the sealing key by specifying ATTRIBUTESMASK and MISCMASK in the KEYREQUEST structure. 


37.7.3, Processor Extended States and ENCLS[ECREATE] 


The ECREATE leaf function of the ENCLS instruction enforces a number of consistency checks described earlier. The 
execution of ENCLS[ECREATE] leaf function results in a #GP(O) in any of the following cases: 


° SECS.ATTRIBUTES.XFRM[1:0] is not 3. 
¢ The processor does not support XSAVE and any of the following is true: 
— SECS.ATTRIBUTES.XFRM[63:2] is not 0. 
— SECS.SSAFRAMESIZE is 0. 
¢ The processor supports XSAVE and any of the following is true: 
— XSETBV would fault on an attempt to load XFRM into XCRO. 
— XFRM[63]=1. 
— The SSAFRAME is too small to hold required, enabled states (see Section 37.7.2.2). 


37.7.4 Processor Extended States and ENCLU[EENTER] 


37.7.4.1 Fault Checking 


The EENTER leaf function of the ENCLU instruction enforces a number of consistency requirements described 
earlier. The execution of the ENCLU[EENTER] leaf function results in a #GP(0) in any of the following cases: 


° If CR4.0SFXSR=0. 

¢ If The processor supports XSAVE and either of the following is true: 
— CR4.0SXSAVE=0 and SECS.ATTRIBUTES.XFRM is not 3. 
— (SECS.ATTRIBUTES.XFRM & XCRO) != SECS.ATTRIBUTES.XFRM 


37.7.4.2 State Loading 


If ENCLU[EENTER] is successful, the current value of XCRO is saved internally by the processor and replaced by 
SECS.ATTRIBUTES.XFRM. 
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37.7.5 Processor Extended States and AEX 


37.7.5.1 State Saving 


On an AEX, processor extended states are saved into the XSAVE area of the SSA frame in a compatible format with 
XSAVE that was executed with EDX:EAX = SECS.ATTRIBUTES.XFRM, with the memory operand being the XSAVE 
area, and (for 64-bit enclaves) as if REX.W=1. The XSTATE_BV part of the XSAVE header is saved with 0 for every 
bit that is 0 in XFRM. Other bits may be saved as 0 if the state saved is initialized. 


Note that enclave entry ensures that if CR4.0SXSAVE is set to 0, then SECS.ATTRIBUTES. XFRM is set to 3. It 
should also be noted that it is not possible to enter an enclave with FXSAVE disabled. 


37.7.5.2 State Synthesis 


After saving the extended state, the processor restores XCRO to the value it held at the time of the most recent 
enclave entry. 


The state of features corresponding to bits set in XFRM is synthesized. In general, these states are initialized. 
Details of state synthesis on AEX are documented in Section 35.3.1. 


37.7.6 Processor Extended States and ENCLU[ERESUME] 


37.7.6.1 Fault Checking 


The ERESUME leaf function of the ENCLU instruction enforces a number of consistency requirements described 
earlier. Specifically, the ENCLU[ERESUME] leaf function results in a #GP(0) in any of the following cases: 


¢ CR4.0SFXSR=0. 
¢ The processor supports XSAVE and either of the following is true: 
— CR4.0SXSAVE=0 and SECS.ATTRIBUTES.XFRM is not 3. 
— (SECS.ATTRIBUTES.XFRM & XCRO) != SECS.ATTRIBUTES.XFRM. 


A successful execution of ENCLU[ERESUME] loads state from the XSAVE area of the SSA frame in a fashion similar 
to that used by the XRSTOR instruction. Data in the XSAVE area that would cause the XRSTOR instruction to fault 
will cause the ENCLU[ERESUME] leaf function to fault. Examples include, but are not restricted to the following: 


e =A bit is set in the XSTATE_BV field and clear in XFRM. 

° The required bytes in the header are not clear. 

® Loading data would set a reserved bit in MXCSR. 

Any of these conditions will cause ERESUME to fault, even if CR4.OSXSAVE=0. 


37.7.6.2 State Loading 


If ENCLU[ERESUME] is successful, the current value of XCRO is saved internally by the processor and replaced by 
SECS.ATTRIBUTES.XFRM. 


State is loaded from the XSAVE area of the SSA frame as if the XRSTOR instruction were executed with 
XCRO=XFRM, EDX:EAX = XFRM, with the memory operand being the XSAVE area, and (for 64-bit enclaves) as if 
REX.W=1. 


ENCLU[ERESUME] ensures that a subsequent execution of XSAVEOPT inside the enclave will operate properly (e.g., 
by marking all state as modified). 


37.7.7. Processor Extended States and ENCLU[EEXIT] 


The ENCLU[EEXIT] leaf function does not perform any X-feature specific consistency checks, nor performs any 
state synthesis. It is the responsibility of enclave software to clear any sensitive data from the registers before 
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executing EEXIT. However, successful execution of the ENCLU[EEXIT] leaf function restores XCRO to the value it 
held at the time of the most recent enclave entry. 


37.7.8 Processor Extended States and ENCLU[EREPORT] 


The ENCLU[EREPORT] leaf function creates the MAC-protected REPORT structure that reports on the enclave’s 
identity. ENCLU[EREPORT] includes in the report the values of SECS.ATTRIBUTES.XFRM and SECS.MISCSELECT. 


37.7.9 Processor Extended States and ENCLU[EGETKEY] 


The ENCLU[EGETKEY] leaf function returns a cryptographic key based on the information provided by the KEYRE- 
QUEST structure. Intel SGX provides the means for isolation between different operating conditions by allowing an 
enclave developer to select which bits out of XFRM and MISCSELECT need to be included in the derivation of the 
keys. 


37.8 INTERACTIONS WITH SMM 


37.8.1 Availability of Intel? SGX instructions in SMM 


Enclave instructions are not available in SMM, and any attempt to execute ENCLS or ENCLU instructions inside SMM 
results in an invalid-opcode exception (#UD). 


37.8.2 SMI while Inside an Enclave 


If the logical processor executing inside an enclave receives an SMI, the logical processor exits the enclave asyn- 
chronously. The response to an SMI received while executing inside an enclave depends on whether the dual- 
monitor treatment is enabled. For detailed discussion of transfer to SMM, see Chapter 30, “System Management 
Mode” of the Inte/® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C. 


If the logical processor executing inside an enclave receives an SMI when dual-monitor treatment is not enabled, 
the logical processor exits the enclave asynchronously, and transfers the control to the SMM handler. In addition to 
saving the synthetic architectural state to the SMRAM State Save Map (SSM), the logical processor also sets the 
“Enclave Interruption” bit in the SMRAM SSM (bit position 1 in SMRAM field at offset 7EEOH). 

If the logical processor executing inside an enclave receives an SMI when dual-monitor treatment is enabled, the 
logical processor exits the enclave asynchronously, and transfers the control to the SMM monitor via SMM VM exit. 
The SMM VM exit sets the “Enclave Interruption” bit in the Exit Reason (see Table 37-2) and in the Guest Interrupt- 
ibility State field (see Table 37-3) of the SMM VMCS. 


37.8.3 SMRAM Synthetic State of AEX Triggered by SMI 


All processor registers saved in the SMRAM have the same synthetic values listed in Section 35.3. Additional 
SMRAM fields that are treated specially on SMI are: 


Table 37-2. SMRAM Synthetic States on Asynchronous Enclave Exit 
Position Field Value Writable 


SMRAM Offset O7EEOH.Bit 1 ENCLAVE_INTERRUPTION | Set to 1 if exit occurred in enclave mode No 
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37.9 INTERACTIONS OF INIT, SIPI, AND WAIT-FOR-SIPI WITH INTEL® SGX 


INIT received inside an enclave, while the logical processor is not in VMX operation, causes the logical processor to 
exit the enclave asynchronously. After the AEX, the processor's architectural state is initialized to “Power-on” state 
(Table 9.1 in Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). If the logical processor 
is BSP, then it proceeds to execute the BIOS initialization code. If the logical processor is an AP, it enters wait-for- 
SIPI state. 


INIT received inside an enclave, while the logical processor (LP) is in VMX root operation, follows regular Intel 
Architecture behavior and is blocked. 


INIT received inside an enclave, while the logical processor is in VMX non-root operation, causes an AEX. Subse- 
quent to the AEX, the INIT causes a VM exit with the Enclave Interruption bit in the exit-reason field in the VMCS. 


A processor cannot be inside an enclave in the wait-for-SIPI state. Consequently, a SIPI received while inside an 
enclave is lost. 


Intel SGX does not change the behavior of the processor in the wait-for-SIPI state. 
The SGX-related processor states after INIT-SIPI-SIPI is as follows: 

¢ EPC Settings: Unchanged 

° EPCM: Unchanged 

° CPUID.LEAF_12H.*: Unchanged 

¢ ENCLAVE_MODE: 0 (LP exits enclave asynchronously) 

° MEE state: Unchanged 


Software should be aware that following INIT-SIPI-SIPI, the EPC might contain valid pages and should take appro- 
priate measures such as initialize the EPC with the EREMOVE leaf function. 


37.10 INTERACTIONS WITH DMA 


DMA is not allowed to access any Processor Reserved Memory. 


37.11 INTERACTIONS WITH TXT 


37.11.1  Enclaves Created Prior to Execution of GETSEC 


Enclaves which have been created before the GETSEC[SENTER] leaf function are available for execution after the 
successful completion of GETSEC[SENTER] and the corresponding SINIT ACM. Actions that a TXT Launched Envi- 
ronment performs in preparation to execute code in the Launched Environment, also applies to enclave code that 
would run after GETSEC[SENTER]. 


37.11.2 Interaction of GETSEC with Intel® SGX 


All leaf functions of the GETSEC instruction are illegal inside an enclave, and results in an invalid-opcode exception 
(#UD). 


Responding Logical Processors (RLP) which are executing inside an enclave at the time a GETSEC[SENTER] event 
occurs perform an AEX from the enclave and then enter the Wait-for-SIPI state. 


RLP executing inside an enclave at the time of GETSEC[SEXIT], behave as defined for GETSEC[SEXIT]-that is, the 
RLPs pause during execution of SEXIT and resume after the completion of SEXIT. 


The execution of a TXT launch does not affect Intel SGX configuration or security parameters. 
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37.11.3 Interactions with Authenticated Code Modules (ACMs) 


Intel SGX only allows launching ACMs with an Intel SGX SVN that is at the same level or higher than the expected 
Intel SGX SVN. The expected Intel SGX SVN is specified by BIOS and locked down by the processor on the first 
successful execution of an Intel SGX instruction that doesn’t return an error code. Intel SGX provides interfaces for 
system software to discover whether a non faulting Intel SGX instruction has been executed, and evaluate the suit- 
ability of the Intel SGX SVN value of any ACM that is expected to be launched by the OS or the VMM. 


These interfaces are provided through a read-only MSR called the IA32_SGX_SVN_STATUS MSR (MSR address 
500h). The IA32_SGX_SVN_STATUS MSR has the format shown in Table 37-3. 


Table 37-3. Layout of the IA32_SGX_SVN_STATUS MSR 
Bit Position Name ACM Module ID Value 


0 Lock N.A. 


If 1, indicates that a non-faulting Intel SGX instruction has been 
executed, consequently, launching a properly signed ACM but with Intel 
SGX SVN value less than the BIOS specified Intel SGX SVN threshold 
would lead to an TXT shutdown. 

If 0, indicates that the processor will allow a properly signed ACM to 
launch irrespective of the Intel SGX SVN value of the ACM. 


15:1 RSVD NA. 0 
23:16 SGX_SVN_SINIT | SINIT ACM * If CPUID.O1H:ECX.SMX =1, this field reflects the expected threshold of 
Intel SGX SVN for the SINIT ACM. 
«If CPUID.O1H:ECX.SMX =0, this field is reserved (0). 
63:24 RSVD NA. 0 


OS/VMM that wishes to launch an architectural ACM such as SINIT is expected to read the IA32_SGX_SVN_STATUS 
MSR to determine whether the ACM can be launched or a new ACM is needed: 


e If either the Intel SGX SVN of the ACM is greater than the value reported by IA32_SGX_SVN_STATUS, or the 
lock bit in the IA32_SGX_SVN_STATUS is not set, then the OS/VMM can safely launch the ACM. 


° If the Intel SGX SVN value reported in the corresponding component of the IA32_SGX_SVN_STATUS is greater 
than the Intel SGX SVN value in the ACM's header, and if bit 0 of IA32_SGX_SVN_STATUS is 1, then the 
OS/VMM should not launch that version of the ACM. It should obtain an updated version of the ACM either from 
the BIOS or from an external resource. 


However, OSVs/VMMs are strongly advised to update their version of the ACM any time they detect that the Intel 
SGX SVN of the ACM carried by the OS/VMM is lower than that reported by IA32_SGX_SVN_STATUS MSR, irrespec- 
tive of the setting of the lock bit. 


37.12 INTERACTIONS WITH CACHING OF LINEAR-ADDRESS TRANSLATIONS 


Entering and exiting an enclave causes the logical processor to flush all the global linear-address context as well as 
the linear-address context associated with the current VPID and PCID. The MONITOR FSM is also cleared. 


37.13. INTERACTIONS WITH INTEL® TRANSACTIONAL SYNCHRONIZATION 
EXTENSIONS (INTEL® TSX) 


1. ENCLU or ENCLS instructions inside an HLE region will cause the flow to be aborted and restarted non-specula- 
tively. ENCLU or ENCLS instructions inside an RTM region will cause the flow to be aborted and transfer control to 
the fallback handler. 


2. If XBEGIN is executed inside an enclave, the processor does NOT check whether the address of the fallback 
handler is within the enclave. 


3. If an RTM transaction is executing inside an enclave and there is an attempt to fetch an instruction outside the 
enclave, the transaction is aborted and control is transferred to the fallback handler. No #GP is delivered. 
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4. If an RTM transaction is executing inside an enclave and there is a data access to an address within the enclave 
that denied due to EPCM content (e.g., to a page belonging to a different enclave), the transaction is aborted and 
control is transferred to the fallback handler. No #GP is delivered. 


5. If an RTM transaction executing inside an enclave aborts and the address of the fallback handler is outside the 
enclave, a #GP is delivered after the abort (EIP reported is that of the fallback handler). 


37.13.1 HLE and RTM Debug 


RTM debug will be suppressed on opt-out enclave entry. After opt-out entry, the logical processor will behave as if 
IA32_DEBUG_CTL[15]=0. Any #DB detected inside an RTM transaction region will just cause an abort with no 
exception delivered. 


After opt-in entry, if either DR7[11] = 0 OR IA32_DEBUGCTL[15] = 0, any #DB or #BP detected inside an RTM 
transaction region will just cause an abort with no exception delivered. 


After opt-in entry, if DR7[11] = 1 AND IA32_DEBUGCTL[15] = 1, any #DB or #BP detected inside an RTM transla- 
tion will 


® terminate speculative execution, 
* set RIP to the address of the XBEGIN instruction, and 
° be delivered as #DB (implying an Intel SGX AEX; any #BP is converted to #DB). 


* DR6[16] will be cleared, indicating RTM debug (if the #DB causes a VM exit, DR6 is not modified but bit 16 of 
the pending debug exceptions field in the VMCS will be set). 


37.14 INTEL® SGX INTERACTIONS WITH S STATES 


Whenever an Intel SGX enabled processor enters S3-S5 state, enclaves are destroyed. This is due to the EPC being 
destroyed when power down occurs. It is the application runtime’s responsibility to re-instantiate an enclave after 
a power transition for which the enclaves were destroyed. 


37.15 INTEL® SGX INTERACTIONS WITH MACHINE CHECK ARCHITECTURE (MCA) 


37.15.1 Interactions with MCA Events 


All architecturally visible machine check events (#MC and CMCI) that are detected while inside an enclave cause an 
asynchronous enclave exit. 


Any machine check exception (#MC) that occurs after Intel SGX is first enables causes Intel SGX to be disabled, 
(CPUID.SGX_Leaf.0: EAX[SGX1] == 0). It cannot be enabled until after the next reset. 


37.15.2 Machine Check Enables (IA32_MCi_CTL) 


All supported IA32_MCi_CTL bits for all the machine check banks must be set for Intel SGX to be available 
(CPUID.SGX_Leaf.0: EAX[SGX1] == 1). Any act of clearing bits from '1 to '0 in any of the IA32_MCi_CTL register 
may disable Intel SGX (set CPUID.SGX_Leaf.0: EAX[SGX1] to 0) until the next reset. 


37.15.3 CR4.MCE 


CR4.MCE can be set or cleared with no interactions with Intel SGX. 
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37.16 INTEL® SGX INTERACTIONS WITH PROTECTED MODE VIRTUAL 
INTERRUPTS 


ENCLS[EENTER] modifies neither EFLAGS.VIP nor EFLAGS.VIF. 


ENCLS[ERESUME] loads EFLAGS in a manner similar to that of an execution of IRET with CPL = 3. This means that 
ERESUME modifies neither EFLAGS.VIP nor EFLAGS.VIF regardless of the value of the EFLAGS image in the SSA 
frame. 

AEX saves EFLAGS.VIP and EFLAGS.VIF unmodified into the EFLAGS image in the SSA frame. AEX modifies neither 
EFLAGS.VIP nor EFLAGS.VIF after saving EFLAGS. 


If CR4.PVI = 1, CPL = 3, EFLAGS.VM = 0, IOPL < 3, EFLAGS.VIP = 1, and EFLAGS.VIF = 0, execution of STI causes 
a #GP fault. In this case, STI modifies neither EFLAGS.IF nor EFLAGS.VIF. This behavior applies without change 
within an enclave (where CPL is always 3). Note that, if IOPL = 3, STI always sets EFLAGS.IF without fault; 
CR4.PVI, EFLAGS.VIP, and EFLAGS.VIF are neither consulted nor modified in this case. 


37.17 INTEL SGX INTERACTION WITH PROTECTION KEYS 


SGX interactions with PKRU are as follows: 


° = CPUID.(EAX=12H, ECX=1):ECX.PKRU indicates whether SECS.ATTRIBUTES.XFRM.PKRU can be set. If 
SECS.ATTRIBUTES.XFRM.PKRU is set, then PKRU is saved and cleared as part of AEX and is restored as part of 
ERESUME. If CR4.PKE is set, an enclave can execute RDPKRU and WRKRU independent of whether 
SECS.ATTRIBUTES.XFRM.PKRU is set. 


SGX interactions with domain permission checks are as follows: 
1) If CR4.PKE is not set, then legacy and SGX permission checks are not effected. 


2) If CR4.PKE is set, then domain permission checks are applied to all non-enclave access and 
enclave accesses to user pages in addition to legacy and SGX permission checks at a higher 
priority than SGX permission checks. 


3) Implicit accesses aren't subject to domain permission checks. 
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CHAPTER 38 
ENCLAVE CODE DEBUG AND PROFILING 


Intel® SGX is architected to provide protection for production enclaves and permit enclave code developers to use 
an SGX-aware debugger to effectively debug a non-production enclave (debug enclave). Intel SGX also allows a 
non-SGX-aware debugger to debug non-enclave portions of the application without getting confused by enclave 
instructions. 


38.1 CONFIGURATION AND CONTROLS 


38.1.1 Debug Enclave vs. Production Enclave 


The SECS of each enclave provides a bit, SECS.ATTRIBUTES.DEBUG, indicating whether the enclave is a debug 
enclave (if set) or a production enclave (if 0). If this bit is set, software outside the enclave can use 
EDBGRD/EDBGWR to access the EPC memory of the enclave. The value of DEBUG is not included in the measure- 
ment of the enclave and therefore doesn't require an alternate SIGSTRUCT to be generated to debug the enclave. 


The ATTRIBUTES field in the SECS is reported in the enclave's attestation, and is included in the key derivation. 
Enclave secrets that were protected by the enclave using Intel SGX keys when it ran as a production enclave will 
not be accessible by the debug enclave. A debugger needs to be aware that special debug content might be 
required for a debug enclave to run in a meaningful way. 

EPC memory belonging to a debug enclave can be accessed via the EDBGRD/EDBGWR leaf functions (see Section 
36.4), while that belonging to a non-debug enclave cannot be accessed by these leaf functions. 


38.1.2 Tool-Chain Opt-in 


The TCS.FLAGS.DBGOPTIN bit controls interactions of certain debug and profiling features with enclaves, including 
code/data breakpoints, TF, RF, monitor trap flag, BTF, LBRs, BTM, BTS, Intel Processor Trace, and performance 
monitoring. This bit is forced to zero when EPC pages are added via EADD. A debugger can set this bit via EDBGWR 
to the TCS of a debug enclave. 


An enclave entry through a TCS with the TCS.FLAGS.DBGOPTIN set to 0 is called an opt-out entry. Conversely, an 
enclave entry through a TCS with TCS.FLAGS.DBGOPTIN set to 1 is called an opt-in entry. 


38.2 SINGLE STEP DEBUG 


38.2.1 Single Stepping ENCLS Instruction Leafs 


If the RFLAGS.TF bit is set at the beginning of ENCLS, then a single-step debug exception is pending as a trap-class 
exception on the instruction boundary immediately after the ENCLS instruction. Additionally, if the instruction is 
executed in VMX non-root operation and the “monitor trap flag” VM-execution control is 1, an MTF VM exit is 
pending on the instruction boundary immediately after the instruction if the instruction does not fault. 


38.2.2 Single Stepping ENCLU Instruction Leafs 
The interactions of the unprivileged Intel SGX instruction ENCLU are leaf dependent. 


An enclave entry via EENTER/ERESUME leaf functions of the ENCLU, in certain cases, may mask the RFLAGS.TF bit, 
and mask the setting of the “monitor trap flag” VM-execution control. In such situations, an exit from the enclave, 
either via the EEXIT leaf function or via an AEX unmasks the RFLAGS.TF bit and the “monitor trap flag” VM-execu- 
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tion control. The details of this masking/unmasking and the pending of single stepping events across 
EENTER/ERESUME/EEXIT/AEX are covered in detail in Section 38.2.3. 


If the EFLAGS.TF bit is set at the beginning of EREPORT or EGETKEY leafs, and if the EFLAGS.TF is not masked by 
the preceding enclave entry, then a single-step debug exception is pending on the instruction boundary immedi- 
ately after the ENCLU instruction. Additionally, if the instruction is executed in VMX non-root operation and the 
“monitor trap flag” VM-execution control is 1, and if the monitor trap flag is not masked by the preceding enclave 
entry, then an MTF VM exit is pending on the instruction boundary immediately after the instruction. 


If the instruction under consideration results in a fault, then the control flow goes to the fault handler, and no 
single-step debug exception is asserted. In such a situation, if the instruction is executed in VMX non-root opera- 
tion and the “monitor trap flag” VM-execution control is 1, an MTF VM exit is pending after the delivery of the fault 
(or any nested exception). No MTF VM exit occurs if another VM exit occurs before reaching that boundary on which 
an MTF VM exit would be pending. 


38.2.3 Single-Stepping Enclave Entry with Opt-out Entry 


38.2.3.1 Single Stepping without AEX 
Figure 38-1 shows the most common case for single-stepping after an opt-out entry. 


p EENTER » 08 
ERESUME = 
) Inst1 ——y|_  Inst2 |-—»|_= Inst =|) EEXIT Inst4 


) _ TF/MTF 


Handler 


SMI 
INIT 
RFLAGS.TF @  sinate-step #08 Pending #MC 
Higher Priority 
VMCS.MTF ()_ MTF VM Exit Pending L — — >} handler 


Figure 38-1. Single Stepping with Opt-out Entry - No AEX 


In this scenario, if the RFLAGS.TF bit is set at the time of the enclave entry, then a single step debug exception is 
pending on the instruction boundary after EEXIT. Additionally, if the enclave is executing in VMX non-root operation 
and the “monitor trap flag” VM-execution control is 1, an MTF VM exit is pending on the instruction boundary after 
EEXIT. 


The value of the RFLAGS.TF bit at the end of EEXIT is the same as the value of RFLAGS.TF at the time of the enclave 
entry. 


38.2.3.2 Single Step Preempted by AEX Due to Non-SMI Event 
Figure 38-2 shows the interaction of single stepping with AEX due to a non-SMI event after an opt-out entry. 
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Figure 38-2. Single Stepping with Opt-out Entry -AEX Due to Non-SMI Event Before Single-Step Boundary 


In this scenario, if the enclave is executing in VMX non-root operation and the “monitor trap flag” VM-execution 
control is 1, an MTF VM exit is pending on the instruction boundary after the AEX. No MTF VM exit occurs if another 
VM exit happens before reaching that instruction boundary. 


The value of the RFLAGS.TF bit at the end of AEX is the same as the value of RFLAGS.TF at the time of the enclave 
entry. 


38.2.4 RFLAGS.TF Treatment on AEX 


The value of EFLAGS.TF at the end of AEX from an opt-out enclave is same as the value of EFLAGS.TF at the time 
of the enclave entry. The value of EFLAGS.TF at the end of AEX from an opt-in enclave is unmodified. The 
EFLAGS.TF saved in GPR portion of the SSA on an AEX is 0. For more detail see EENTER and ERESUME in Chapter 5. 


38.2.5 Restriction on Setting of TF after an Opt-Out Entry 


Enclave entered through an opt-out entry is not allowed to set EFLAGS.TF. The POPF instruction forces RFLAGS.TF 
to O if the enclave was entered through opt-out entry. 


38.2.6 Trampoline Code Considerations 


Any AEX from the enclave which results in the RFLAGS.TF =1 on the reporting stack will result in a single-step #DB 
after the first instruction of the trampoline code if the trampoline is entered using the IRET instruction. 


38.3. CODE AND DATA BREAKPOINTS 


38.3.1 Breakpoint Suppression 

Following an opt-out entry: 

¢ Instruction breakpoints are suppressed during execution in an enclave. 

° Data breakpoints are not triggered on accesses to the address range defined by ELRANGE. 
¢ Data breakpoints are triggered on accesses to addresses outside the ELRANGE 
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Following an opt-in entry instruction and data breakpoints are not suppressed. 


The processor does not report any matches on debug breakpoints that are suppressed on enclave entry. However, 
the processor does not clear any bits in DR6 that were already set at the time of the enclave entry. 


38.3.2 Reporting of Instruction Breakpoint on Next Instruction on a Debug Trap 


A debug exception caused by the single-step execution mode or when a data breakpoint condition was met causes 
the processor to perform an AEX. Following such an AEX, the processor reports in the debug status register (DR6) 
matches of the new instruction pointer (the AEP address) in a breakpoint address register setup to detect instruc- 
tion execution. 


38.3.3. RF Treatment on AEX 


RF flag value saved in SSA is the same as what would have been pushed on stack if the exception or event causing 
the AEX occurred when executing outside an enclave (see Section 17.3.1.1). Following an AEX, the RF flag is 0 in 
the synthetic state. 


38.3.4 Breakpoint Matching in Intel® SGX Instruction Flows 


Implicit accesses made by Intel SGX instructions to EPC regions do not trigger data breakpoints. Explicit accesses 
made by ENCLS[ECREATE], ENCLS[EADD], ENCLS[EEXTEND], ENCLS[EINIT], ENCLS[EREMOVE], 
ENCLS[ETRACK], ENCLS[EBLOCK], ENCLS[EPA], ENCLS[EWB], ENCLS[ELD], ENCLS[EDBGRD], ENCLS[EDBGWR], 
ENCLU[EENTER], and ENCLU[ERESUME] to the EPC operands do not trigger data breakpoints. 


Explicit accesses made by the Intel SGX instructions (ENCLU[EGETKEY] and ENCLU[EREPORT]) executed by an 
enclave following an opt-in entry, trigger data breakpoints on accesses to their EPC operands. All Intel SGX instruc- 
tions trigger data breakpoints on accesses to their non-EPC operands. 


38.4 CONSIDERATION OF THE INT1 AND INT3 INSTRUCTIONS 


This section considers the operation of the INT1 and INT3 instructions when executed inside an enclave. These are 
the instructions with opcodes F1 and CC, respectively, and not INT n (with opcode CD) with value 1 or 3 for n. 


38.4.1 Behavior of INT1 and INT3 Inside an Enclave 


An execution of either INT1 or INT3 inside an enclave results in a fault-class exception. Following an opt-out entry, 
execution of either instruction results in an invalid-opcode exception (#UD). Following opt-in entry, INT1 results in 
a debug exception (#DB) and INT3 delivers a breakpoint exception (#BP). The normal requirement for INT3 (that 
the CPL not be greater than the DPL of descriptor 3 in the IDT) is not enforced. 


Because execution of INT1 or INT3 inside an enclave results in a fault, the RIP saved in the SSA on AEX references 
the INT1 or INT3 instruction (and not the following instruction). The RIP value saved on the stack (or in the TSS or 
VMCS) is that of the AEP. 


If execution of INT1 or INT3 inside an enclave causes a VM exit, the event type in the VM-exit interruption informa- 
tion field indicates a hardware exception (type 3),! and the VM-exit instruction length field is saved as zero. 


38.4.2 Debugger Considerations 


A debugger using INT3 inside an enclave should account for the modified behavior described in Section 38.4.1. 
Because INTS3 is fault-like inside an enclave, the RIP saved in the SSA on AExX is that of the INT3 instruction. Conse- 


1. INT1 would normally indicate a privileged software exception (type 5), and INT3 would normally indicate a software exception (type 6). 
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quently, the debugger must not decrement SSA.RIP for #BP coming from an enclave to re-execute the instruction 
at the RIP of the INT3 instruction on a subsequent enclave entry. 


38.4.3. VMM Considerations 


As described in Section 38.4.1, execution of INT3 inside an enclave delivers #BP with “interruption type” of 3. A 
VMM that re-injects #BP into the guest should establish the VM-entry interruption information field using data 
saved into the appropriate VMCS fields by the VM exit incident to the #BP (as recommended in Inte/® 64 and 
IA-32 Architectures Software Developer’s Manual, Volume 3C). 


VMMs that create the VM-entry interruption information based solely on the exception vector should take care to 
use event type 3 (instead of 6) when they detect a VM exit incident to enclave mode that is due to an exception 
with vector 3. 


38.5 BRANCH TRACING 


38.5.1 BTF Treatment 


When software enables single-stepping on branches then: 
¢ Following an opt-in entry using EENTER the processor generates a single step debug exception. 
¢ Following an EEXIT the processor generates a single-step debug exception 


Enclave entry using ERESUME (opt-in or opt-out) and an AEX from the enclave do not cause generation of the 
single-step debug exception. 


38.5.2 LBR Treatment 


38.5.2.1_ LBR Stack on Opt-in Entry 


Following an opt-in entry into an enclave, last branch recording facilities if enabled continued to store branch 
records in the LBR stack MSRs as follows: 


° On enclave entry using EENTER/ERESUME, the processor push the address of EENTER/ERESUME instruction 
into MSR_LASTBRANCH_n_FROM_IP, and the destination address of the EENTER/ERESUME into 
MSR_LASTBRANCH_n_TO_ IP. 


¢ On EEXIT, the processor pushes the address of EEXIT instruction into MSR_LASTBRANCH_n_FROM_IP, and the 
address of EEXIT destination into MSR_LASTBRANCH_n_TO_IP. 


* On AEX, the processor pushes RIP saved in the SSA into MSR_LASTBRANCH_n_FROM_IP, and the address of 
AEP into MSR_LASTBRANCH_n_TO_ IP. 


¢ For every branch inside the enclave, a branch record is pushed on the LBR stack. 


Figure 38-3 shows an example of LBR stack manipulation after an opt-in entry. Every arrow in this picture indicates 
a branch record pushed on the LBR stack. The “From IP” of the branch record contains the linear address of the 
instruction located at the start of the arrow, while the “To IP” of the branch record contains the linear address of the 
instruction at the end of the arrow. 
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Figure 38-3. LBR Stack Interaction with Opt-in Entry 


38.5.2.2 LBR Stack on Opt-out Entry 


An opt-out entry into an enclave suppresses last branch recording facilities, and enclave exit after an opt-out entry 
un-suppresses last branch recording facilities. 


Opt-out entry into an enclave does not push any record on LBR stack. 


If last branch recording facilities were enabled at the time of enclave entry, then EEXIT following such an enclave 
entry pushes one record on LBR stack. The MSR_LASTBRANCH_n_FROM_IP of such record holds the linear address 
of the instruction (EENTER or ERESUME) that was used to enter the enclave, while the 
MSR_LASTBRANCH_n_TO_IP of such record holds linear address of the destination of EEXIT. 


Additionally, if last branch recording facilities were enabled at the time of enclave entry, then an AEX after such an 
entry pushes one record on LBR stack, before pushing record for the event causing the AEX if the event pushes a 
record on LBR stack. The MSR_LASTBRANCH_n_FROM_IP of the new record holds linear address of the instruction 
(EENTER or ERESUME) that was used to enter the enclave, while MSR_LASTBRANCH_n_TO_IP of the new record 
holds linear address of the AEP. If the event causing AEX pushes a record on LBR stack, then the 
MSR_LASTBRANCH_n_FROM_IP for that record holds linear address of the AEP. 


Figure 38-4 shows an example of LBR stack manipulation after an opt-out entry. Every arrow in this picture indi- 
cates a branch record pushed on the LBR stack. The “From IP” of the branch record contains the linear address of 
the instruction located at the start of the arrow, while the “To IP” of the branch record contains the linear address 
of the instruction at the end of the arrow. 
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Figure 38-4. LBR Stack Interaction with Opt-out Entry 


38.5.2.3 Mispredict Bit, Record Type, and Filtering 


All branch records resulting from Intel SGX instructions/AEXs are reported as predicted branches, and conse- 
quently, bit 63 of MSR_LASTBRANCH_n_FROM_IP for such records is set. Branch records due to these Intel SGX 
operations are always non-HLE/non-RTM records. 


EENTER, ERESUME, EEXIT, and AEX are considered to be far branches. Consequently, bit 8 in MSR_LBR_SELECT 
controls filtering of the new records introduced by Intel SGX. 


38.6 INTERACTION WITH PERFORMANCE MONITORING 


38.6.1 IA32_PERF_GLOBAL_STATUS Enhancement 


On processors supporting Intel SGX, the IA32_PERF_GLOBAL_STATUS MSR provides a bit indicator, known as “Anti 
Side-channel Interference” (ASCI) at bit position 60. If this bit is 0, the performance monitoring data in various 
performance monitoring counters are accumulated normally as defined by relevant architectural/microarchitec- 
tural conditions. If the ASCI bit is set, the contents in various performance monitoring counters can be affected by 
the direct or indirect consequence of Intel SGX protection of enclave code executing in the processor. 


38.6.2 Performance Monitoring with Opt-in Entry 


An opt-in enclave entry allow performance monitoring logic to observe the contribution of enclave code executing 
in the processor. Thus the contents of performance monitoring counters does not distinguish between contribution 
originating from enclave code or otherwise. All counters, events, precise events, etc. continue to work as defined 
in the IA32/Intel 64 Software Developer Manual. Consequently, bit 60 of IA32_PERF_GLOBAL_STATUS MSR is not 
set. 
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38.6.3. Performance Monitoring with Opt-out Entry 


In general, performance monitoring activities are suppressed when entering an opt-out enclave. This applies to all 
thread-specific, configured performance monitoring, except for the cycle-counting fixed counter, 
IA32_FIXED_CTR1 and IA32_FIXED_CTR2. Upon entering an opt-out enclave, IA32_FIXED_CTRO, IA32_PMCx will 
stop accumulating counts. Additionally, if PEBS is configured to capture PEBS record for this thread, PEBS record 
generation will also be suppressed. Consequently, bit 60 of IA32_PERF_GLOBAL_STATUS MSR is set. 


Performance monitoring on the sibling thread may also be affected. Any one of IA32_FIXED_CTRx or IA32_PMCx 
on the sibling thread configured to monitor thread-specific eventing logic with AnyThread =1 is demoted to count 
only MyThread while an opt-out enclave is executing on the other thread. 


38.6.4 Enclave Exit and Performance Monitoring 


When a logical processor exits an enclave, either via ENCLU[EEXIT] or via AEX, all performance monitoring activity 
(including PEBS) on that logical processor that was suppressed is unsuppressed. 


Any counters that were demoted from AnyThread to MyThread on the sibling thread are promoted back to 
AnyThread. 


38.6.5 PEBS Record Generation on Intel® SGX Instructions 


All leaf functions of the ENCLS instruction report “Eventing RIP” of the ENCLS instruction if a PEBS record is gener- 
ated at the end of the instruction execution. Additionally, the EGETKEY and EREPORT leaf functions of the ENCLU 
instruction report “Eventing RIP” of the ENCLU instruction if a PEBS record is generated at the end of the instruction 
execution. 


If the EENTER and ERESUME leaf functions are performing an opt-in entry report “Eventing RIP” of the ENCLU 
instruction if a PEBS record is generated at the end of the instruction execution. On the other hand, if these leaf 
functions are performing an opt-out entry, then these leaf functions result in PEBS being suppressed, and no PEBS 
record is generated at the end of these instructions. 


A PEBS record is generated if there is a PEBS event pending at the end of EEXIT (due to a counter overflowing 
during enclave execution or during EEXIT execution). This PEBS record contains the architectural state of the 
logical processor at the end of EEXIT. If the enclave was entered via an opt-in entry, then this record reports the 
“Eventing RIP” as the linear address of the ENCLU[EEXIT] instruction. If the enclave was entered via an opt-out 
entry, then the record reports the “Eventing RIP” as the linear address of the ENCLU[EENTER/ERESUME] instruc- 
tion that performed the last enclave entry. 


A PEBS record is generated after the AEX if there is a PEBS event pending at the end of AEX (due to a counter over- 
flowing during enclave execution or during AEX execution). This PEBS record contains the synthetic state of the 
logical processor that is established at the end of AEX. For opt-in entry, this record has the EVENTING_RIP set to 
the RIP saved in the SSA. For opt-out entry, the record has the EVENTING_RIP set to the linear address of 
EENTER/ERESUME used for the last enclave entry. 


If the enclave was entered via an opt-in entry, then this record reports the “Eventing RIP” as the linear address in 
the SSA of the enclave (a.k.a., the “Eventing LIP” inside the enclave). If the enclave was entered via an opt-out 
entry, then the record reports the “Eventing RIP” as the linear address of the ENCLU[EENTER/ERESUME] instruc- 
tion that performed the last enclave entry. 


A second PEBS event may be pended during the Enclave Exiting Event (EEE). If the PEBS event is taken at the end 
of delivery of the EEE then the “Eventing RIP” in this second PEBS record is the linear address of the AEP. 


38.6.6 Exception-Handling on PEBS/BTS Loads/Stores after AEX 


As noted in Section 17.4.9.2, recording in the BTS buffer or in the PEBS buffer may not operate properly if accesses 
to any of the DS save area sections cause page faults or VM exits. Such page faults or VM exits, if they occur, are 
delivered immediately to the OS or VMM, and generation of a BTS or PEBS record is skipped and may leave the 
buffers in a state where they have a partial BTS or PEBS records. 


However, any events that are detected during PEBS/BTS record generation at the end of AEX and before delivering 
the Enclave Exiting Event (EEE) cannot be reported immediately to the OS/VMM, as an event window is not open at 


38-8 Vol. 3D 


ENCLAVE CODE DEBUG AND PROFILING 


the end of AEX. Consequently, fault-like events such as page faults, EPT faults, EPT mis-configuration, and 
accesses to APIC-access page detected on stores to the PEBS/BTS buffer are not reported, and generation of the 
PEBS and/or BTS record at the end of AEX is aborted (this may leave the buffers in a state where they have partial 
PEBS or BTS records). Trap-like events detected on stores to the PEBS/BTS buffer (such as debug traps) are 
pended until the next instruction boundary, where they are handled according to the architecturally defined 
priority. The processor continues the handling of the Enclave Exiting Event (SMI, NMI, interrupt, exception delivery, 
VM exit, etc.) after aborting the PEBS/BTS record generation. 


38.6.6.1 Other Interactions with Performance Monitoring 


For opt-in entry, EENTER, ERESUME, EEXIT, and AEX are all treated as predicted far branches, and any counters 
that are counting such branches are incremented by 1 as a part of retirement of these instructions. Retirement of 
these instructions is also counted in any counters configured to count instructions retired. 


For opt-out entry, execution inside an enclave is treated as a single predicted branch, and all branch-counting 
performance monitoring counters are incremented accordingly. Additionally, such execution is also counted as a 
single instruction, and all performance monitoring counters counting instructions are incremented accordingly. 


Enclave entry does not affect any performance monitoring counters shared between cores. 
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APPENDIX A 
VMX CAPABILITY REPORTING FACILITY 


The ability of a processor to support VMX operation and related instructions is indicated by 
CPUID.1:ECX.VMX[bit 5] = 1. A value 1 in this bit indicates support for VMX features. 


Support for specific features detailed in Chapter 25 and other VMX chapters is determined by reading values from 
a set of capability MSRs. These MSRs are indexed starting at MSR address 480H. VMX capability MSRs are read- 
only; an attempt to write them (with WRMSR) produces a general-protection exception (#GP(0)). They do not exist 
on processors that do not support VMX operation; an attempt to read them (with RDMSR) on such processors 
produces a general-protection exception (#GP(0)). 


A.1 BASIC VMX INFORMATION 


The IA32_VMX_BASIC MSR (index 480H) consists of the following fields: 


¢ Bits 30:0 contain the 31-bit VMCS revision identifier used by the processor. Processors that use the same VMCS 
revision identifier use the same size for VMCS regions (see subsequent item on bits 44:32).4 


° = -Bit 31 is always 0. 


° Bits 44:32 report the number of bytes that software should allocate for the VMXON region and any VMCS 
region. It is a value greater than 0 and at most 4096 (bit 44 is set if and only if bits 43:32 are clear). 


¢ Bit 48 indicates the width of the physical addresses that may be used for the VMXON region, each VMCS, and 
data structures referenced by pointers in a VMCS (I/O bitmaps, virtual-APIC page, MSR areas for VMX transi- 
tions). If the bit is 0, these addresses are limited to the processor’s physical-address width.? If the bit is 1, 
these addresses are limited to 32 bits. This bit is always 0 for processors that support Intel 64 architecture. 


° If bit 49 is read as 1, the logical processor supports the dual-monitor treatment of system-management 
interrupts and system-management mode. See Section 30.15 for details of this treatment. 


¢ Bits 53:50 report the memory type that should be used for the VMCS, for data structures referenced by 
pointers in the VMCS (I/O bitmaps, virtual-APIC page, MSR areas for VMX transitions), and for the MSEG 
header. If software needs to access these data structures (e.g., to modify the contents of the MSR bitmaps), it 
can configure the paging structures to map them into the linear-address space. If it does so, it should establish 
mappings that use the memory type reported bits 53:50 in this MSR.? 


As of this writing, all processors that support VMX operation indicate the write-back type. The values used are 
given in Table A-1. 


Table A-1. Memory Types Recommended for VMCS and Related Data Structures 


Value(s) Field 

0 Uncacheable (UC) 
1-5 Not used 

6 Write Back (WB) 
7-15 Not used 


1. Earlier versions of this manual specified that the VMCS revision identifier was a 32-bit field in bits 31:0 of this MSR. For all proces- 
sors produced prior to this change, bit 31 of this MSR was read as 0. 


2. On processors that support Intel 64 architecture, the pointer must not set bits beyond the processor's physical address width. 


3. Alternatively, software may map any of these regions or structures with the UC memory type. (This may be necessary for the MSEG 
header.) Doing so is discouraged unless necessary as it will cause the performance of software accesses to those structures to suf- 
fer. 
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° If bit 54 is read as 1, the processor reports information in the VM-exit instruction-information field on VM exits 
due to execution of the INS and OUTS instructions (see Section 26.2.5). This reporting is done only if this bit is 
read as 1. 

° =Bit 55 is read as 1 if any VMX controls that default to 1 may be cleared to 0. See Appendix A.2 for details. It also 
reports support for the VMX capability MSRs IA32_VMX_TRUE_PINBASED_CTLS, 
IA32_VMX_TRUE_PROCBASED_CTLS, IA32_VMX_TRUE_EXIT_CTLS, and IA32_VMX_TRUE_ENTRY_CTLS. See 
Appendix A.3.1, Appendix A.3.2, Appendix A.4, and Appendix A.5 for details. 

e If bit 56 is read as 1, software can use VM entry to deliver a hardware exception with or without an error code, 
regardless of vector (see Section 25.2.1.3). 


¢ The values of bits 47:45 and bits 63:57 are reserved and are read as 0. 


A.2 RESERVED CONTROLS AND DEFAULT SETTINGS 


As noted in Chapter 25, “VM Entries”, certain VMX controls are reserved and must be set to a specific value (0 or 1) 
determined by the processor. The specific value to which a reserved control must be set is its default setting. 
Software can discover the default setting of a reserved control by consulting the appropriate VMX capability MSR 
(see Appendix A.3 through Appendix A.5). 


Future processors may define new functionality for one or more reserved controls. Such processors would allow 
each newly defined control to be set either to 0 or to 1. Software that does not desire a control’s new functionality 
should set the control to its default setting. For that reason, it is useful for software to know the default settings of 
the reserved controls. 


Default settings partition the various controls into the following classes: 

¢ Always-flexible. These have never been reserved. 

° DefaultO. These are (or have been) reserved with a default setting of 0. 
* Default1. They are (or have been) reserved with a default setting of 1. 


As noted in Appendix A.1, a logical processor uses bit 55 of the IA32_VMX_BASIC MSR to indicate whether any of 

the default1 controls may be 0: 

° If bit 55 of the IA32_VMX_BASIC MSR is read as 0, all the defaulti controls are reserved and must be 1. 
VM entry will fail if any of these controls are 0 (see Section 25.2.1). 

es Tf bit 55 of the IA32_VMX_BASIC MSR is read as 1, not all the default1 controls are reserved, and some (but 
not necessarily all) may be 0. The CPU supports four (4) new VMX capability MSRs: 
IA32_VMX_TRUE_PINBASED_CTLS, IA32_VMX_TRUE_PROCBASED_CTLS, IA32_VMX_TRUE_EXIT_CTLS, and 
IA32_VMX_TRUE_ENTRY_CTLS. See Appendix A.3 through Appendix A.5 for details. (These MSRs are not 
supported if bit 55 of the IA32_VMX_BASIC MSR is read as 0.) 


A.3 VM-EXECUTION CONTROLS 


There are separate capability MSRs for the pin-based VM-execution controls, the primary processor-based VM- 
execution controls, the secondary processor-based VM-execution controls, and the tertiary processor-based VM- 
execution controls. These are described in Appendix A.3.1, Appendix A.3.2, Appendix A.3.3, and Appendix A.3.4, 
respectively. 


A.3.1 Pin-Based VM-Execution Controls 


The IA32_VMX_PINBASED_CTLS MSR (index 481H) reports on the allowed settings of most of the pin-based 
VM-execution controls (see Section 23.6.1): 


¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X (bit X of the pin-based 
VM-execution controls) to be 0 if bit X in the MSR is cleared to 0; if bit X in the MSR is set to 1, VM entry fails if 
control X is 0. 
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Exceptions are made for the pin-based VM-execution controls in the default1 class (see Appendix A.2). These 
are bits 1, 2, and 4; the corresponding bits of the IA32_VMX_PINBASED_CTLS MSR are always read as 1. The 
treatment of these controls by VM entry is determined by bit 55 in the IA32_VMX_BASIC MSR: 


— If bit 55 in the IA32_VMX_BASIC MSR is read as 0, VM entry fails if any pin-based VM-execution control in 
the default1 class is 0. 


— If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_PINBASED_CTLS MSR (see 
below) reports which of the pin-based VM-execution controls in the default1 class can be 0 on VM entry. 


¢ Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control X to be 1 if bit 32+X in 
the MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 


If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_PINBASED_CTLS MSR (index 48DH) 
reports on the allowed settings of all of the pin-based VM-execution controls: 


¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X to be 0 if bit X in the MSR 
is cleared to 0; if bit X in the MSR is set to 1, VM entry fails if control X is 0. There are no exceptions. 


¢ Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control X to be 1 if bit 32+ X in the 
MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 


It is necessary for software to consult only one of the capability MSRs to determine the allowed settings of the pin- 
based VM-execution controls: 


¢ = If bit 55 in the IA32_VMX_BASIC MSR is read as 0, all information about the allowed settings of the pin-based 
VM-execution controls is contained in the IA32_VMX_PINBASED_CTLS MSR. (The 
IA32_VMX_TRUE_PINBASED_CTLS MSR is not supported. ) 


¢ = If bit 55 in the IA32_VMX_BASIC MSR is read as 1, all information about the allowed settings of the pin-based 
VM-execution controls is contained in the IA32_VMX_TRUE_PINBASED_CTLS MSR. Assuming that software 
knows that the default1 class of pin-based VM-execution controls contains bits 1, 2, and 4, there is no need for 
software to consult the IA32_VMX_PINBASED_CTLS MSR. 


A.3.2 Primary Processor-Based VM-Execution Controls 


The IA32_VMX_PROCBASED_CTLS MSR (index 482H) reports on the allowed settings of most of the primary 
processor-based VM-execution controls (see Section 23.6.2): 


° Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X (bit X of the primary 
processor-based VM-execution controls) to be 0 if bit X in the MSR is cleared to 0; if bit X in the MSR is set to 
1, VM entry fails if control X is 0. 


Exceptions are made for the primary processor-based VM-execution controls in the defaulti class (see 
Appendix A.2). These are bits 1, 4-6, 8, 13-16, and 26; the corresponding bits of the 
IA32_VMX_PROCBASED_CTLS MSR are always read as 1. The treatment of these controls by VM entry is 
determined by bit 55 in the IA32_VMX_BASIC MSR: 


— If bit 55 in the IA32_VMX_BASIC MSR is read as 0, VM entry fails if any of the primary processor-based VM- 
execution controls in the defaulti class is 0. 


— If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_PROCBASED_CTLS MSR (see 
below) reports which of the primary processor-based VM-execution controls in the default1 class can be 0 
on VM entry. 


¢ Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control X to be 1 if bit 32+ X in the 
MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 


If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_PROCBASED_CTLS MSR (index 48EH) 
reports on the allowed settings of all of the primary processor-based VM-execution controls: 


¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X to be 0 if bit X in the MSR 
is cleared to 0; if bit X in the MSR is set to 1, VM entry fails if control X is 0. There are no exceptions. 


° Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control X to be 1 if bit 32+ X in the 
MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 


It is necessary for software to consult only one of the capability MSRs to determine the allowed settings of the 
primary processor-based VM-execution controls: 
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e = Tf bit 55 in the IA32_VMX_BASIC MSR is read as 0, all information about the allowed settings of the primary 
processor-based VM-execution controls is contained in the IA32_VMX_PROCBASED_CTLS MSR. (The 
IA32_VMX_TRUE_PROCBASED_CTLS MSR is not supported.) 

e = Tf bit 55 in the IA32_VMX_BASIC MSR is read as 1, all information about the allowed settings of the processor- 
based VM-execution controls is contained in the IA32_VMX_TRUE_PROCBASED_CTLS MSR. Assuming that 
software knows that the default1 class of processor-based VM-execution controls contains bits 1, 4-6, 8, 13- 
16, and 26, there is no need for software to consult the IA32_VMX_PROCBASED_CTLS MSR. 


A.3.3 Secondary Processor-Based VM-Execution Controls 


The IA32_VMX_PROCBASED_CTLS2 MSR (index 48BH) reports on the allowed settings of the secondary processor- 
based VM-execution controls (see Section 23.6.2). The following items provide details, including enforcement by 
VM entry: 


¢ Bits 31:0 indicate the allowed O-settings of these controls. These bits are always 0. This fact indicates that 
VM entry allows each bit of the secondary processor-based VM-execution controls to be 0 (reserved bits must 
be 0) 

° Bits 63:32 indicate the allowed i-settings of these controls; the 1-setting is not allowed for any reserved bit. 
VM entry allows control X (bit X of the secondary processor-based VM-execution controls) to be 1 if bit 32+X in 
the MSR is set to 1; if bit 32+ X in the MSR is cleared to 0, VM entry fails if control X and the “activate secondary 
controls” primary processor-based VM-execution control are both 1. 


The IA32_VMX_PROCBASED_CTLS2 MSR exists only on processors that support the 1-setting of the “activate 
secondary controls” VM-execution control (only if bit 63 of the IA32_VMX_PROCBASED_CTLS MSR is 1). 


A.3.4 Tertiary Processor-Based VM-Execution Controls 


The IA32_VMX_PROCBASED_CTLS3 MSR (index 492H) reports on the allowed 1-settings of the tertiary processor- 
based VM-execution controls (see Section 23.6.2); the 1-setting is not allowed for any reserved bit. 


VM entry allows control X (bit X of the tertiary processor-based VM-execution controls) to be 1 if bit X in the MSR 
is set to 1; if bit X in the MSR is cleared to 0, VM entry fails if control X and the “activate tertiary controls” primary 
processor-based VM-execution control are both 1. 


The IA32_VMX_PROCBASED_CTLS3 MSR exists only on processors that support the 1-setting of the “activate 
tertiary controls” VM-execution control (only if bit 49 of the IA32_VMX_PROCBASED_CTLS MSR is 1). 


Notice that the organization of this MSR differs from that of IA32_VMX_PROCBASED_CTLS2 (Appendix A.3.3). This 
is because there are 64 tertiary processor-based VM-execution controls, while there were only 32 secondary 
processor-based VM-execution controls. 


A.4 VM-EXIT CONTROLS 
The IA32_VMX_EXIT_CTLS MSR (index 483H) reports on the allowed settings of most of the VM-exit controls (see 
Section 23.7.1): 


¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X (bit X of the VM-exit 
controls) to be 0 if bit X in the MSR is cleared to 0; if bit X in the MSR is set to 1, VM entry fails if control X is 0. 


Exceptions are made for the VM-exit controls in the default1 class (see Appendix A.2). These are bits 0-8, 10, 
11, 13, 14, 16, and 17; the corresponding bits of the IA32_VMX_EXIT_CTLS MSR are always read as 1. The 
treatment of these controls by VM entry is determined by bit 55 in the IA32_VMX_BASIC MSR: 


— If bit 55 inthe IA32_VMX_BASIC MSR is read as 0, VM entry fails if any VM-exit control in the default class 
is 0. 

— If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_EXIT_CTLS MSR (see below) 
reports which of the VM-exit controls in the default1 class can be 0 on VM entry. 


° Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control 32+X to be 1 if bit X in the 
MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 
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If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_EXIT_CTLS MSR (index 48FH) reports on 

the allowed settings of all of the VM-exit controls: 

¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X to be 0 if bit X in the MSR 
is cleared to O; if bit X in the MSR is set to 1, VM entry fails if control X is 0. There are no exceptions. 


° Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control X to be 1 if bit 32+ X in the 
MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 


It is necessary for software to consult only one of the capability MSRs to determine the allowed settings of the 

VM-exit controls: 

¢ = If bit 55 in the IA32_VMX_BASIC MSR is read as 0, all information about the allowed settings of the VM-exit 
controls is contained in the IA32_VMX_EXIT_CTLS MSR. (The IA32_VMX_TRUE_EXIT_CTLS MSR is not 
supported.) 

¢ If bit 55 in the IA32_VMX_BASIC MSR is read as 1, all information about the allowed settings of the VM-exit 
controls is contained in the IA32_VMX_TRUE_EXIT_CTLS MSR. Assuming that software knows that the default1 
class of VM-exit controls contains bits 0-8, 10, 11, 13, 14, 16, and 17, there is no need for software to consult 
the IA32_VMX_EXIT_CTLS MSR. 


A.5 VM-ENTRY CONTROLS 


The IA32_VMX_ENTRY_CTLS MSR (index 484H) reports on the allowed settings of most of the VM-entry controls 

(see Section 23.8.1): 

¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X (bit X of the VM-entry 
controls) to be 0 if bit X in the MSR is cleared to 0; if bit X in the MSR is set to 1, VM entry fails if control X is 0. 
Exceptions are made for the VM-entry controls in the default1 class (see Appendix A.2). These are bits 0-8 and 
12; the corresponding bits of the IA32_VMX_ENTRY_CTLS MSR are always read as 1. The treatment of these 
controls by VM entry is determined by bit 55 in the IA32_VMX_BASIC MSR: 


— If bit 55 in the IA32_VMX_BASIC MSR is read as 0, VM entry fails if any VM-entry control in the default1 
class is 0. 
— If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_ENTRY_CTLS MSR (see below) 
reports which of the VM-entry controls in the defaulti class can be 0 on VM entry. 
¢ = Bits 63:32 indicate the allowed 1-settings of these controls. VM entry fails if bit X is 1 in the VM-entry controls 
and bit 32+ X is 0 in this MSR. 
If bit 55 in the IA32_VMX_BASIC MSR is read as 1, the IA32_VMX_TRUE_ENTRY_CTLS MSR (index 490H) reports 
on the allowed settings of all of the VM-entry controls: 


¢ Bits 31:0 indicate the allowed O-settings of these controls. VM entry allows control X to be 0 if bit X in the MSR 
is cleared to 0; if bit X in the MSR is set to 1, VM entry fails if control X is 0. There are no exceptions. 


¢ Bits 63:32 indicate the allowed 1-settings of these controls. VM entry allows control 32+ X to be 1 if bit X in the 
MSR is set to 1; if bit 32+X in the MSR is cleared to 0, VM entry fails if control X is 1. 


It is necessary for software to consult only one of the capability MSRs to determine the allowed settings of the 

VM-entry controls: 

¢ If bit 55 in the IA32_VMX_BASIC MSR is read as 0, all information about the allowed settings of the VM-entry 
controls is contained in the IA32_VMX_ENTRY_CTLS MSR. (The IA32_VMX_TRUE_ENTRY_CTLS MSR is not 
supported.) 

¢ If bit 55 in the IA32_VMX_BASIC MSR is read as 1, all information about the allowed settings of the VM-entry 
controls is contained in the IA32_VMX_TRUE_ENTRY_CTLS MSR. Assuming that software knows that the 
defaulti class of VM-entry controls contains bits O-8 and 12, there is no need for software to consult the 
IA32_VMX_ENTRY_CTLS MSR. 
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A.6 MISCELLANEOUS DATA 


The IA32_VMX_MISC MSR (index 485H) consists of the following fields: 


Bits 4:0 report a value X that specifies the relationship between the rate of the VMX-preemption timer and that 
of the timestamp counter (TSC). Specifically, the VMX-preemption timer (if it is active) counts down by 1 every 
time bit X in the TSC changes due to a TSC increment. 

If bit 5 is read as 1, VM exits store the value of IA32_EFER.LMA into the “IA-32e mode guest” VM-entry control; 
see Section 26.2 for more details. This bit is read as 1 on any logical processor that supports the 1-setting of 
the “unrestricted guest” VM-execution control. 


Bits 8:6 report, as a bitmap, the activity states supported by the implementation: 
— Bit 6 reports (if set) the support for activity state 1 (HLT). 

— Bit 7 reports (if set) the support for activity state 2 (shutdown). 

— Bit 8 reports (if set) the support for activity state 3 (wait-for-SIPI). 


If an activity state is not supported, the implementation causes a VM entry to fail if it attempts to establish that 
activity state. All implementations support VM entry to activity state O (active). 


If bit 14 is read as 1, Intel® Processor Trace (Intel PT) can be used in VMX operation. If the processor supports 
Intel PT but does not allow it to be used in VMX operation, execution of VMXON clears IA32_RTIT_CTL.TraceEn 
(see “VMXON—Enter VMX Operation” in Chapter 29); any attempt to write IA32_RTIT_CTL while in VMX 
operation (including VMX root operation) causes a general-protection exception. 


If bit 15 is read as 1, the RDMSR instruction can be used in system-management mode (SMM) to read the 
IA32_SMBASE MSR (MSR address 9EH). See Section 30.15.6.3. 

Bits 24:16 indicate the number of CR3-target values supported by the processor. This number is a value 
between 0 and 256, inclusive (bit 24 is set if and only if bits 23:16 are clear). 


Bits 27:25 is used to compute the recommended maximum number of MSRs that should appear in the VM-exit 
MSR-store list, the VM-exit MSR-load list, or the VM-entry MSR-load list. Specifically, if the value bits 27:25 of 
IA32_VMX_MISC is N, then 512 * (N + 1) is the recommended maximum number of MSRs to be included in 
each list. If the limit is exceeded, undefined processor behavior may result (including a machine check during 
the VMX transition). 


If bit 28 is read as 1, bit 2 of the IA32_SMM_MONITOR_CTL can be set to 1. VMXOFF unblocks SMIs unless 
IA32_SMM_MONITOR_CTL[bit 2] is 1 (see Section 30.14.4). 


If bit 29 is read as 1, software can use VMWRITE to write to any supported field in the VMCS; otherwise, 
VMWRITE cannot be used to modify VM-exit information fields. 


If bit 30 is read as 1, VM entry allows injection of a software interrupt, software exception, or privileged 
software exception with an instruction length of 0. 


Bits 63:32 report the 32-bit MSEG revision identifier used by the processor. 
Bits 13:9 and bit 31 are reserved and are read as 0. 


A.7 VMX-FIXED BITS IN CRO 


The IA32_VMX_CRO_FIXEDO MSR (index 486H) and IA32_VMX_CRO_FIXED1 MSR (index 487H) indicate how bits 
in CRO may be set in VMX operation. They report on bits in CRO that are allowed to be 0 and to be 1, respectively, 
in VMX operation. If bit X is 1 in IA32_VMX_CRO_FIXEDO, then that bit of CRO is fixed to 1 in VMX operation. Simi- 
larly, if bit X is O in IA32_VMX_CRO_FIXED1, then that bit of CRO is fixed to 0 in VMX operation. It is always the case 
that, if bit X is 1 in IA32_VMX_CRO_FIXEDO, then that bit is also 1 in IA32_VWMX_CRO_FIXED1; if bit X is O in 
IA32_VMX_CRO_FIXED1, then that bit is also 0 in IA32_VMX_CRO_FIXEDO. Thus, each bit in CRO is either fixed to 
0 (with value 0 in both MSRs), fixed to 1 (1 in both MSRs), or flexible (0 in IA32_VMX_CRO_FIXEDO and 1 in 
IA32_VMX_CRO_FIXED1). 
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A.8 VMX-FIXED BITS IN CR4 


The IA32_VMX_CR4_FIXEDO MSR (index 488H) and IA32_VMX_CR4_FIXED1 MSR (index 489H) indicate how bits 
in CR4 may be set in VMX operation. They report on bits in CR4 that are allowed to be 0 and 1, respectively, in VMX 
operation. If bit X is 1 in IA32_VWMX_CR4_FIXEDO, then that bit of CR4 is fixed to 1 in VMX operation. Similarly, if 
bit X is 0 in IA32_VMX_CR4_FIXED1, then that bit of CR4 is fixed to 0 in VMX operation. It is always the case that, 
if bit X is 1 in IA32_VMX_CR4_FIXEDO, then that bit is also 1 in IA32_VWMX_CR4_FIXED1; if bit X is O in 
IA32_VMX_CR4_FIXED1, then that bit is also 0 in IA32_VMX_CR4_FIXEDO. Thus, each bit in CR4 is either fixed to 
0 (with value 0 in both MSRs), fixed to 1 (1 in both MSRs), or flexible (0 in IA32_VMX_CR4_FIXEDO and 1 in 
IA32_VMX_CR4_FIXED1). 


A.9 VMCS ENUMERATION 
The IA32_VMX_VMCS_ENUM MSR (index 48AH) provides information to assist software in enumerating fields in 
the VMCS. 


As noted in Section 23.11.2, each field in the VMCS is associated with a 32-bit encoding which is structured as 
follows: 


¢  =6Bits 31:15 are reserved (must be 0). 

° Bits 14:13 indicate the field’s width. 

°  ~=Bit 12 is reserved (must be 0). 

¢ Bits 11:10 indicate the field’s type. 

° Bits 9:1 is an index field that distinguishes different fields with the same width and type. 
¢ Bit 0 indicates access type. 


IA32_VMX_VMCS_ENUM indicates to software the highest index value used in the encoding of any field supported 
by the processor: 


¢ Bits 9:1 contain the highest index value used for any VMCS encoding. 
¢ Bit 0 and bits 63:10 are reserved and are read as O. 


A.10  VPID AND EPT CAPABILITIES 


The IA32_VMX_EPT_VPID_CAP MSR (index 48CH) reports information about the capabilities of the logical 
processor with regard to virtual-processor identifiers (VPIDs, Section 27.1) and extended page tables (EPT, Section 
27.2): 


¢ If bit 0 is read as 1, the processor supports execute-only translations by EPT. This support allows software to 
configure EPT paging-structure entries in which bits 1:0 are clear (indicating that data accesses are not 
allowed) and bit 2 is set (indicating that instruction fetches are allowed). 


° Bit 6 indicates support for a page-walk length of 4. 


¢ If bit 8 is read as 1, the logical processor allows software to configure the EPT paging-structure memory type 
to be uncacheable (UC); see Section 23.6.11. 


¢ If bit 14 is read as 1, the logical processor allows software to configure the EPT paging-structure memory type 
to be write-back (WB). 


¢ = If bit 16 is read as 1, the logical processor allows software to configure a EPT PDE to map a 2-Mbyte page (by 
setting bit 7 in the EPT PDE). 


¢ If bit 17 is read as 1, the logical processor allows software to configure a EPT PDPTE to map a 1-Gbyte page (by 
setting bit 7 in the EPT PDPTE). 


1. If the “mode-based execute control for EPT” VM-execution control is 1, setting bit O indicates also that software may also configure 
EPT paging-structure entries in which bits 1:0 are both clear and in which bit 10 is set (indicating a translation that can be used to 
fetch instructions from a supervisor-mode linear address or a user-mode linear address). 
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° Support for the INVEPT instruction (see Chapter 29 and Section 27.3.3.1). 
— If bit 20 is read as 1, the INVEPT instruction is supported. 
— If bit 25 is read as 1, the single-context INVEPT type is supported. 
— If bit 26 is read as 1, the all-context INVEPT type is supported. 
e Tf bit 21 is read as 1, accessed and dirty flags for EPT are supported (see Section 27.2.5). 


° If bit 22 is read as 1, the processor reports advanced VM-exit information for EPT violations (see Section 
26.2.1). This reporting is done only if this bit is read as 1. 


e If bit 23 is read as 1, supervisor shadow-stack control is supported (see Section 27.2.3.2). 
¢ Support for the INVVPID instruction (see Chapter 29 and Section 27.3.3.1). 
— If bit 32 is read as 1, the INVVPID instruction is supported. 
— If bit 40 is read as 1, the individual-address INVVPID type is supported. 
— If bit 41 is read as 1, the single-context INVVPID type is supported. 
— If bit 42 is read as 1, the all-context INVVPID type is supported. 
— If bit 43 is read as 1, the single-context-retaining-globals INVVPID type is supported. 
°® Bits 5:1, bit 7, bits 13:9, bit 15, bits 19:18, bit 24, bits 31:27, bits 39:33, and bits 63:44 are reserved and are 
read as 0. 


The IA32_VMX_EPT_VPID_CAP MSR exists only on processors that support the 1-setting of the “activate secondary 
controls” VM-execution control (only if bit 63 of the IA32_VMX_PROCBASED_CTLS MSR is 1) and that support 
either the 1-setting of the “enable EPT” VM-execution control (only if bit 33 of the IA32_VMX_PROCBASED_CTLS2 
MSR is 1) or the 1-setting of the “enable VPID” VM-execution control (only if bit 37 of the 
IA32_VMX_PROCBASED_CTLS2 MSR is 1). 


A.11 WMFUNCTIONS 


The IA32_VMX_VMFUNC MSR (index 491H) reports on the allowed settings of the VM-function controls (see 
Section 23.6.14). VM entry allows bit X of the VM-function controls to be 1 if bit X in the MSR is set to 1; if bit X in 
the MSR is cleared to 0, VM entry fails if bit X of the VM-function controls, the “activate secondary controls” primary 
processor-based VM-execution control, and the “enable VM functions” secondary processor-based VM-execution 
control are all 1. 


The IA32_VMX_VMFUNC MSR exists only on processors that support the 1-setting of the “activate secondary 
controls” VM-execution control (only if bit 63 of the IA32_VMX_PROCBASED_CTLS MSR is 1) and the 1-setting of 
the “enable VM functions” secondary processor-based VM-execution control (only if bit 45 of the 
IA32_VMX_PROCBASED_CTLS2 MSR is 1). 
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Every component of the VMCS is encoded by a 32-bit field that can be used by VMREAD and VMWRITE. Section 
23.11.2 describes the structure of the encoding space (the meanings of the bits in each 32-bit encoding). 


This appendix enumerates all fields in the VMCS and their encodings. Fields are grouped by width (16-bit, 32-bit, 
etc.) and type (guest-state, host-state, etc.) 


B.1 16-BIT FIELDS 


A value of 0 in bits 14:13 of an encoding indicates a 16-bit field. Only guest-state areas and the host-state area 
contain 16-bit fields. As noted in Section 23.11.2, each 16-bit field allows only full access, meaning that bit O of its 
encoding is 0. Each such encoding is thus an even number. 


B.1.1 16-Bit Control Fields 


A value of 0 in bits 11:10 of an encoding indicates a control field. These fields are distinguished by their index value 
in bits 9:1. Table B-1 enumerates the 16-bit control fields. 


Table B-1. Encoding for 16-Bit Control Fields (Q000_O00xx_xxxx_xxx0B) 


Field Name Index Encoding 
Virtual-processor identifier (VPID)! OO00000000B OO0000000H 
Posted-interrupt notification vector@ 000000001B 00000002H 
EPTP index? 000000010B 00000004H 
NOTES: 


1. This field exists only on processors that support the 1-setting of the “enable VPID” VM-execution control. 
2. This field exists only on processors that support the 1-setting of the “process posted interrupts” VM-execution control. 
3. This field exists only on processors that support the 1-setting of the “EPT-violation #VE"” VM-execution control. 


B.1.2 16-Bit Guest-State Fields 


A value of 2 in bits 11:10 of an encoding indicates a field in the guest-state area. These fields are distinguished by 
their index value in bits 9:1. Table B-2 enumerates 16-bit guest-state fields. 


Table B-2. Encodings for 16-Bit Guest-State Fields (Q000_10xx_xxxx_xxx0B) 


Field Name Index Encoding 

Guest ES selector 000000000B 00000800H 
Guest CS selector 000000001B 00000802H 
Guest SS selector 000000010B 00000804H 
Guest DS selector 000000011B 00000806H 
Guest FS selector 000000100B 00000808H 
Guest GS selector 000000101B 0000080AH 
Guest LDTR selector 000000110B 0000080CH 
Guest TR selector 000000111B O0000080EH 
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Table B-2. Encodings for 16-Bit Guest-State Fields (Q000_10xx_xxxx_xxx0B) (Contd.) 


Field Name Index Encoding 
Guest interrupt status! 000001000B 00000810H 
PML index? 000001001B 0000081 2H 
NOTES: 


1. This field exists only on processors that support the 1-setting of the “virtual-interrupt delivery” VM-execution control. 


2. This field exists only on processors that support the 1-setting of the “enable PML” VM-execution control. 


B.1.3 16-Bit Host-State Fields 


A value of 3 in bits 11:10 of an encoding indicates a field in the host-state area. These fields are distinguished by 


their index value in bits 9:1. Table B-3 enumerates the 16-bit host-state fields. 


Table B-3. Encodings for 16-Bit Host-State Fields (Q000_11xx_xxxx_xxx0B) 


Field Name Index Encoding 

Host ES selector 000000000B OOOO0COOH 
Host CS selector 000000001B 00000C02H 
Host SS selector 000000010B 00000C04H 
Host DS selector 000000011B OOOOOCO6H 
Host FS selector 000000100B 00000C08H 
Host GS selector 000000101B OOO00COAH 
Host TR selector 0000001 10B OOOO00COCH 


B.2 64-BIT FIELDS 


A value of 1 in bits 14:13 of an encoding indicates a 64-bit field. There are 64-bit fields only for controls and for 
guest state. As noted in Section 23.11.2, every 64-bit field has two encodings, which differ on bit 0, the access 
type. Thus, each such field has an even encoding for full access and an odd encoding for high access. 


B.2.1 64-Bit Control Fields 


A value of 0 in bits 11:10 of an encoding indicates a control field. These fields are distinguished by their index value 
in bits 9:1. Table B-4 enumerates the 64-bit control fields. 


Table B-4. Encodings for 64-Bit Control Fields (0010_O00xx_xxxx_xxxAb) 


Field Name Index Encoding 
Address of I/O bitmap A (full) 00002000H 
= HTT —#_—¥#__—_———— 0000000008 
Address of I/O bitmap A (high) 00002001H 
Address of I/O bitmap B (full) 00002002H 
000000001B 
Address of 1/0 bitmap B (high) 00002003H 
Address of MSR bitmaps (full)! 00002004H 
000000010B 
Address of MSR bitmaps (high)! 00002005H 
VM-exit MSR-store address (full) 00002006H 
= ——— 00000001 1B 
\V/M-exit MSR-store address (high) 00002007H 
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Table B-4. Encodings for 64-Bit Control Fields (0010_O0xx_xxxx_xxxAb) (Contd.) 


Field Name Index Encoding 

V/M-exit MSR-load address (full) 00002008H 
000000100B 

\V/M-exit MSR-load address (high) 0000200SH 

VM-entry MSR-load address (full) 0000200AH 
000000101B 

VM-entry MSR-load address (high) 0000200BH 

Executive-VMCS pointer (full) 0000200CH 
000000110B 

Executive-VMCS pointer (high) 0000200DH 

PML address (full)* 0000200EH 
ae 000000111B 

PML address (high) 0000200FH 

TSC offset (full) 00002010H 
000001000B 

TSC offset (high) 00002011H 

Virtual-APIC address (full)? 00002012H 
- aa 000001001B 

Virtual-APIC address (high) 00002013H 

APIC-access address (full)* 00002014H 
000001010B 

APIC-access address (high)* 00002015H 

Posted-interrupt descriptor address (full)? 00002016H 
000001011B 

Posted-interrupt descriptor address (high)? 00002017H 

VM-function controls (full)® 00002018H 
000001100B 

VM-function controls (high)® 0000201SH 

EPT pointer (EPTP; full)” 0000201AH 
000001101B 

EPT pointer (EPTP; high)” 0000201BH 

EOI-exit bitmap O (EOI_EXITO; full)® 0000201CH 
000001110B 

EOI-exit bitmap 0 (EOI_EXITO; high)® 0000201DH 

EOl-exit bitmap 1 (EOI_EXIT1; full)® 0000201EH 
— Se 000001111B 

EOI-exit bitmap 1 (EOI_EXIT1; high) 0000201FH 

EOl-exit bitmap 2 (EOI_EXIT2; full)® 00002020H 
— 5 000010000B 

EOI-exit bitmap 2 (EOI_EXIT2; high) 00002021H 

EOl-exit bitmap 3 (EOI_EXIT3; full)® 00002022H 
000010001B 

EOI-exit bitmap 3 (EOI_EXIT3; high)® 00002023H 

EPTP-list address (full)? 00002024H 
- er 000010010B 

EPTP-list address (high) 00002025H 

VMREAD-bitmap address (full)!® 00002026H 
000010011B 

VMREAD-bitmap address (high)!° 00002027H 

VMWRITE-bitmap address (full)!® 00002028H 
000010100B 

VMWRITE-bitmap address (high)!° 00002029SH 

Virtualization-exception information address (full)! ! 0000202AH 
000010101B 

Virtualization-exception information address (high)! | 0000202BH 

XSS-exiting bitmap (full) '2 0000202CH 
—— ie 000010110B 

XSS-exiting bitmap (high) 0000202DH 
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Table B-4. Encodings for 64-Bit Control Fields (0010_O00xx_xxxx_xxxAb) (Contd.) 


Field Name Index Encoding 
ENCLS-exiting bitmap (full)'? 0000202EH 
——- ECE 000010111B 
ENCLS-exiting bitmap (high) 0000202FH 
Sub-page-permission-table pointer (full)'4 00002030H 
00001 1000B 
Sub-page-permission-table pointer (high)'4 00002031H 
TSC multiplier (full)'® 00002032H 
000011001B 
TSC multiplier (high)'® 00002033H 
Tertiary processor-based /M-execution controls (full)'® dae rinide 00002034H 
Tertiary processor-based VM-execution controls (high)'® 00002035H 
ENCLV-exiting bitmap (full)!” 00002036H 
——— a7 000011011B 
ENCLV-exiting bitmap (high) 00002037H 


NOTES: 

1. This field exists only on processors that support the 1-setting of the 
2. This field exists only on processors that support the 1-setting of the 
3. This field exists only on processors that support the 1-setting of the 
4. This field exists only on processors that support the 1-setting of the 
5. This field exists only on processors that support the 1-setting of the 
6. This field exists only on processors that support the 1-setting of the 
7. This field exists only on processors that support the 1-setting of the 
8. This field exists only on processors that support the 1-setting of the 
9. This field exists only on processors that support the 1-setting of the 
10. This field exists only on processors that support the 1-setting of the 
11. This field exists only on processors that support the 1-setting of the 
12. This field exists only on processors that support the 1-setting of the 
13. This field exists only on processors that support the 1-setting of the 
14. This field exists only on processors that support the 1-setting of the 
15. This field exists only on processors that support the 1-setting of the 
16. This field exists only on processors that support the 1-setting of the 
17. This field exists only on processors that support the 1-setting of the 


B.2.2 64-Bit Read-Only Data Field 


“use MSR bitmaps” VM-execution control. 


“enable PML” VM-execution control. 

“use TPR shadow” VM-execution control. 
“virtualize APIC accesses” VM-execution control. 
“process posted interrupts” VM-execution control. 
“enable VM functions” VM-execution control. 
“enable EPT” VM-execution control. 
“virtual-interrupt delivery” VM-execution control. 
“EPTP switching” VM-function control. 


am 


VMCS shadowing” VM-execution control. 

“EPT-violation #VE" VM-execution control. 

“enable XSAVES/XRSTORS” VM-execution control. 

“enable ENCLS exiting” VM-execution control. 

“sub-page write permissions for EPT” VM-execution control. 
“use TSC scaling” VM-execution control. 

“activate tertiary controls” VM-execution control. 

“enable ENCLV exiting” VM-execution control. 


A value of 1 in bits 11:10 of an encoding indicates a read-only data field. These fields are distinguished by their 


index value in bits 9:1. There is only one such 64-bit field as g 
has two encodings. ) 


iven in Table B-5.(As with other 64-bit fields, this one 


Table B-5. Encodings for 64-Bit Read-Only Data Field (0010_01xx_xxxx_xxxAb) 


Field Name Index Encoding 

Guest-physical address (full)! 00002400H 
O000000000B 

Guest-physical address (high)! 00002401H 


NOTES: 


1. This field exists only on processors that support the 1-setting of the ” 
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enable EPT” VM-execution control. 


B.2.3 64-Bit Guest-State Fields 


FIELD ENCODING IN VMCS 


A value of 2 in bits 11:10 of an encoding indicates a field in the guest-state area. These fields are distinguished by 
their index value in bits 9:1. Table B-6 enumerates the 64-bit guest-state fields. 


Table B-6. Encodings for 64-Bit Guest-State Fields (0010_10xx_xxxx_xxxAb) 


Field Name Index Encoding 

VMCS link pointer (full) 00002800H 
000000000B 

VMCS link pointer (high) 00002801H 

Guest IA32_DEBUGCTL (full) 00002802H 
000000001B 

Guest IA32_DEBUGCTL (high) 00002803H 

Guest IA32_PAT (full)! 00002804H 
000000010B 

Guest IA32_PAT (high)! 00002805H 

Guest |A32_EFER (full)? 00002806H 
ae 000000011B 

Guest IA32_EFER (high) 00002807H 

Guest IA32_PERF_GLOBAL_CTRL (full)? 00002808H 
eC 000000100B 

Guest IA32_PERF_GLOBAL_CTRL (high) 00002809H 

Guest PDPTEO (full)* 0000280AH 
000000101B 

Guest PDPTEO (high)* 0000280BH 

Guest PDPTE1 (full)* 0000280CH 
000000110B 

Guest PDPTE1 (high)* 0000280DH 

Guest PDPTE2 (full)* 0000280EH 
000000111B 

Guest PDPTE2 (high)* 0000280FH 

Guest PDPTE3 (full)* 00002810H 
000001000B 

Guest PDPTE3 (high)* 00002811H 

Guest I[A32_BNDCFGS (full)? 00002812H 
000001001B 

Guest IA32_BNDCFGS (high)? 00002813H 

Guest IA32_RTIT_CTL (full)® 00002814H 
6 000001010B 

Guest IA32_RTIT_CTL (high) 00002815H 

Guest IA32_PKRS (full)” 00002818H 
000001 100B 

Guest IA32_PKRS (high)” 00002819H 


NOTES: 


1. This field exists only on processors that support either the 1-setting of the “load IA32_PAT” VM-entry control or that of the “save 


IA32_PAT” VM-exit control. 


2. This field exists only on processors that support either the 1-setting of the “load IA32_EFER” VM-entry control or that of the “save 


IA32_EFER” VM-exit control. 


3. This field exists only on processors that support the 1-setting of the “load IA32_PERF_GLOBAL_CTRL” VM-entry control. 


4. This field exists only on processors that support the 1-setting of the “enable EPT” VM-execution control. 


5. This field exists only on processors that support either the 1-setting of the “load [A32_BNDCFGS” VM-entry control or that of the 


“clear [A32_BNDCFGS” VM-exit control. 


6. This field exists only on processors that support either the 1-setting of the “load IA32_RTIT_CTL” VM-entry control or that of the 


“clear IA32_RTIT_CTL” VM-exit control. 


7. This field exists only on processors that support the 1-setting of the “load PKRS” VM-entry control. 
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B.2.4 64-Bit Host-State Fields 


A value of 3 in bits 11:10 of an encoding indicates a field in the host-state area. These fields are distinguished by 
their index value in bits 9:1. Table B-7 enumerates the 64-bit control fields. 


Table B-7. Encodings for 64-Bit Host-State Fields (0010_11xx_xxxx_xxxAb) 


Field Name Index Encoding 
Host IA32_PAT (full)! 00002C00H 
000000000B 
Host IA32_PAT (high)! 00002C01H 
Host IA32_EFER (full)? 00002C02H 
ean 000000001B 
Host IA32_EFER (high) 00002C03H 
Host IA32_PERF_GLOBAL_CTRL (full)? 00002C04H 
000000010B 
Host |A32_PERF_GLOBAL_CTRL (high)? 00002C05H 
Host IA32_PKRS (full)* 00002CO6H 
000000011B 
Host IA32_PKRS (high)* 00002C07H 


NOTES: 

1. This field exists only on processors that support the 1-setting of the “load [A32_PAT” VM-exit control. 

2. This field exists only on processors that support the 1-setting of the “load [A32_EFER” VM-exit control. 

3. This field exists only on processors that support the 1-setting of the “load [A32_PERF_GLOBAL_CTRL” VM-exit control. 
4. This field exists only on processors that support the 1-setting of the “load PKRS” VM-exit control. 


B.3 32-BIT FIELDS 


A value of 2 in bits 14:13 of an encoding indicates a 32-bit field. As noted in Section 23.11.2, each 32-bit field 
allows only full access, meaning that bit 0 of its encoding is 0. Each such encoding is thus an even number. 


B.3.1 32-Bit Control Fields 


A value of 0 in bits 11:10 of an encoding indicates a control field. These fields are distinguished by their index value 
in bits 9:1. Table B-8 enumerates the 32-bit control fields. 


Table B-8. Encodings for 32-Bit Control Fields (0100_O0xx_xxxx_xxx0B) 


Field Name Index Encoding 

Pin-based VM-execution controls 0000000008 00004000H 
Primary processor-based VM-execution controls 000000001B 00004002H 
Exception bitmap 000000010B 00004004H 
Page-fault error-code mask 000000011B 00004006H 
Page-fault error-code match 000000100B 00004008H 
CR3-target count 000000101B 0000400AH 
VM-exit controls 0000001 10B 0000400CH 
VM-exit MSR-store count 000000111B 0000400EH 
VM-exit MSR-load count 000001000B 00004010H 
V/M-entry controls 000001001B 0000401 2H 
VM-entry MSR-load count 000001010B 00004014H 
V/M-entry interruption-information field 000001011B 00004016H 
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Table B-8. Encodings for 32-Bit Control Fields (0100_00xx_xxxx_xxx0B) (Contd.) 


Field Name Index Encoding 
\V/M-entry exception error code 000001100B 00004018H 
\V/M-entry instruction length 000001101B 0000401AH 
TPR threshold! 000001110B 0000401CH 
Secondary processor-based VM-execution controls@ 000001111b 0000401EH 
PLE_Gap? 000010000b 00004020H 
PLE_Window? 000010001b 00004022H 
NOTES: 


1. This field exists only on processors that support the 1-setting of the “use TPR shadow” VM-execution control. 
2. This field exists only on processors that support the 1-setting of the “activate secondary controls” VM-execution control. 
3. This field exists only on processors that support the 1-setting of the “PAUSE-loop exiting” VM-execution control. 


B.3.2 32-Bit Read-Only Data Fields 


A value of 1 in bits 11:10 of an encoding indicates a read-only data field. These fields are distinguished by their 


index value in bits 9:1. Table B-9 enumerates the 32-bit read-only data fields. 


Table B-9. Encodings for 32-Bit Read-Only Data Fields (0100_01xx_xxxx_xxx0B) 


Field Name Index Encoding 

VM-instruction error 0000000008 00004400H 
Exit reason 000000001B 00004402H 
\V/M-exit interruption information 000000010B 00004404H 
\/M-exit interruption error code 000000011B 00004406H 
IDT-vectoring information field 000000100B 00004408H 
IDT-vectoring error code 000000101B 0000440AH 
\V/M-exit instruction length 0000001 10B 0000440CH 
VM-exit instruction information 000000111B 0000440€H 


B.3.3 32-Bit Guest-State Fields 


A value of 2 in bits 11:10 of an encoding indicates a field in the guest-state area. These fields are distinguished by 


their index value in bits 9:1. Table B-10 enumerates the 32-bit guest-state fields. 


Table B-10. Encodings for 32-Bit Guest-State Fields (0100_10xx_xxxx_xxxOB) 


Field Name Index Encoding 

Guest ES limit 000000000B 00004800H 
Guest CS limit 000000001B 00004802H 
Guest SS limit 000000010B 00004804H 
Guest DS limit 000000011B 00004806H 
Guest FS limit 0000001 00B 00004808H 
Guest GS limit 000000101B 0000480AH 
Guest LDTR limit 0000001 10B 0000480CH 
Guest TR limit 000000111B 0000480EH 
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Table B-10. Encodings for 32-Bit Guest-State Fields (0100_10xx_xxxx_xxxOB) (Contd.) 


Field Name Index Encoding 

Guest GDTR limit 000001000B 00004810H 
Guest IDTR limit 000001001B 00004812H 
Guest ES access rights 000001010B 00004814H 
Guest CS access rights 000001011B 00004816H 
Guest SS access rights 000001 100B 00004818H 
Guest DS access rights 000001101B 0000481AH 
Guest FS access rights 000001110B 0000481CH 
Guest GS access rights 000001111B 0000481€H 
Guest LDTR access rights 000010000B 00004820H 
Guest TR access rights 000010001B 00004822H 
Guest interruptibility state 000010010B 00004824H 
Guest activity state 000010011B 00004826H 
Guest SMBASE 000010100B 00004828H 
Guest IA32_SYSENTER_CS 000010101B 0000482AH 
\V/MX-preemption timer value! 000010111B 0000482EH 


NOTES: 


1. This field exists only on processors that support the 1-setting of the “activate VMX-preemption timer” VM-execution control. 


The limit fields for GDTR and IDTR are defined to be 32 bits in width even though these fields are only 16-bits wide 
in the Intel 64 and IA-32 architectures. VM entry ensures that the high 16 bits of both these fields are cleared to 0. 


B.3.4 32-Bit Host-State Field 
A value of 3 in bits 11:10 of an encoding indicates a field in the host-state area. There is only one such 32-bit field 
as given in Table B-11. 


Table B-11. Encoding for 32-Bit Host-State Field (0100_11xx_xxxx_xxx0B) 
Field Name Index Encoding 


Host IA32_SYSENTER_CS O00000000B 00004COOH 


B.4 NATURAL-WIDTH FIELDS 


A value of 3 in bits 14:13 of an encoding indicates a natural-width field. As noted in Section 23.11.2, each of these 
fields allows only full access, meaning that bit O of its encoding is 0. Each such encoding is thus an even number. 


B.4.1 Natural-Width Control Fields 


A value of 0 in bits 11:10 of an encoding indicates a control field. These fields are distinguished by their index value 
in bits 9:1. Table B-12 enumerates the natural-width control fields. 


Table B-12. Encodings for Natural-Width Control Fields (01 10_O00xx_xxxx_xxx0B) 
Field Name Index Encoding 


CRO guest/host mask O000000000B OO0006000H 
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Table B-12. Encodings for Natural-Width Control Fields (01 10_O0xx_xxxx_xxxOB) (Contd.) 


Field Name Index Encoding 

CR4 guest/host mask 000000001B 00006002H 
CRO read shadow 000000010B 00006004H 
CR4 read shadow 000000011B OO006006H 
CR3-target value 0 000000100B 00006008H 
CR3-target value 1 000000101B OOO0600AH 
CR3-target value 2 0000001 10B OO00600CH 
CR3-target value 3! 000000111B OOO0600EH 


NOTES: 


1. If a future implementation supports more than 4 CR3-target values, they will be encoded consecutively following the 4 encodings 


given here. 


B.4.2 Natural-width Read-Only Data Fields 


A value of 1 in bits 11:10 of an encoding indicates a read-only data field. These fields are distinguished by their 


index value in bits 9:1. Table B-13 enumerates the natural-width read-only data fields. 


Table B-13. Encodings for Natural-Width Read-Only Data Fields (0110_01xx_xxxx_xxx0OB) 


Field Name Index Encoding 

Exit qualification 000000000B 00006400H 
1/0 RCX 000000001B 00006402H 
I/O RSI 000000010B 00006404H 
I/O RDI 000000011B 00006406H 
I/O RIP 0000001 00B 00006408H 
Guest-linear address 000000101B 0000640AH 


B.4.3 Natural-Width Guest-State Fields 


A value of 2 in bits 11:10 of an encoding indicates a field in the guest-state area. These fields are distinguished by 


their index value in bits 9:1. Table B-14 enumerates the natural-width guest-state fields. 


Table B-14. Encodings for Natural-Width Guest-State Fields (01 10_10xx_xxxx_xxxOB) 


Field Name Index Encoding 

Guest CRO 000000000B 00006800H 
Guest CR3 000000001B 00006802H 
Guest CR4 000000010B 00006804H 
Guest ES base 000000011B 00006806H 
Guest CS base 0000001 00B 00006808H 
Guest SS base 000000101B O0000680AH 
Guest DS base 0000001 10B 0000680CH 
Guest FS base 000000111B O0000680EH 
Guest GS base 000001000B 00006810H 
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Table B-14. Encodings for Natural-Width Guest-State Fields (0110_10xx_xxxx_xxxOB) (Contd.) 


Field Name Index Encoding 
Guest LDTR base 000001001B 00006812H 
Guest TR base 000001010B 00006814H 
Guest GDTR base 000001011B 00006816H 
Guest IDTR base 000001100B 00006818H 
Guest DR7 000001101B 0000681 AH 
Guest RSP 000001110B 0000681CH 
Guest RIP 000001111B 0000681 EH 
Guest RFLAGS 000010000B 00006820H 
Guest pending debug exceptions 000010001B 00006822H 
Guest IA32_SYSENTER_ESP 000010010B 00006824H 
Guest IA32_SYSENTER_EIP 000010011B 00006826H 
Guest IA32_S_CET! 000010100B 00006828H 
Guest SSP! 000010101B 0000682AH 
Guest IA32_INTERRUPT_SSP_TABLE_ADDR' 000010110B 0000682CH 
NOTES: 


1. This field is supported only on processors that support the 1-setting of the “load CET state” VM-entry control. 
The base-address fields for ES, CS, SS, and DS in the guest-state area are defined to be natural-width (with 64 bits 


on processors supporting Intel 64 architecture) even though these fields are only 32-bits wide in the Intel 64 archi- 
tecture. VM entry ensures that the high 32 bits of these fields are cleared to 0. 


B.4.4 Natural-Width Host-State Fields 


A value of 3 in bits 11:10 of an encoding indicates a field in the host-state area. These fields are distinguished by 
their index value in bits 9:1. Table B-15 enumerates the natural-width host-state fields. 


Table B-15. Encodings for Natural-width Host-State Fields (0110_11xx_xxxx_xxxOB) 


Field Name Index Encoding 

Host CRO 000000000B OOOO6COOH 
Host CR3 000000001B 00006C02H 
Host CR4 000000010B O0006C04H 
Host FS base 000000011B OOOO6CO6H 
Host GS base 000000100B OO006CO08H 
Host TR base 000000101B OOOO06COAH 
Host GDTR base 000000110B OOOO6COCH 
Host IDTR base 000000111B OOOO6COEH 
Host IA32_SYSENTER_ESP 000001000B 00006C10H 
Host IA32_SYSENTER_EIP 000001001B 00006C12H 
Host RSP 000001010B 00006C14H 
Host RIP 000001011B 00006C16H 
Host IA32_S_CET! 000001 100B 00006C18H 
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Table B-15. Encodings for Natural-Width Host-State Fields (0110_11xx_xxxx_xxx0B) (Contd.) 


Field Name Index Encoding 
Host SSP! 000001101B 00006C1AH 
Host IA32_INTERRUPT_SSP_TABLE_ADDR' 000001110B 00006C1CH 


NOTES: 


1. This field is supported only on processors that support the 1-setting of the “load CET state” VM-exit control. 
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APPENDIX C 
VMX BASIC EXIT REASONS 


Every VM exit writes a 32-bit exit reason to the VMCS (see Section 23.9.1). Certain VM-entry failures also do this 
(see Section 25.8). The low 16 bits of the exit-reason field form the basic exit reason which provides basic informa- 
tion about the cause of the VM exit or VM-entry failure. 


Table C-1 lists values for basic exit reasons and explains their meaning. Entries apply to VM exits, unless otherwise 
noted. 


Table C-1. Basic Exit Reasons 


Basic Exit 
Reason Description 
0 Exception or non-maskable interrupt (NMI). Either: 
1: Guest software caused an exception and the bit in the exception bitmap associated with exception’s vector was 
1. This case includes executions of BOUND that cause #BR, executions of INT1 (they cause #DB), executions of 
INT3 (they cause #BP), executions of INTO that cause #0F, and executions of UDO, UD1, and UD2 (they cause 
#UD). 
2: An NMI was delivered to the logical processor and the “NMI exiting” VM-execution control was 1. 
1 External interrupt. An external interrupt arrived and the “external-interrupt exiting” VM-execution control was 1. 
2 Triple fault. The logical processor encountered an exception while attempting to call the double-fault handler and 
that exception did not itself cause a VM exit due to the exception bitmap. 
3 INIT signal. An INIT signal arrived 


Start-up IPI (SIPI). A SIPI arrived while the logical processor was in the “wait-for-SIPI” state. 


1/0 system-management interrupt (SMI). An SMI arrived immediately after retirement of an I/O instruction and 
caused an SMM VM exit (See Section 30.15.2). 


6 Other SMI. An SMI arrived and caused an SMM VM exit (see Section 30.15.2) but not immediately after retirement of 
an I/O instruction. 


[7 «ST Interrupt window. At the beginning of an instruction, RFLAGS.IF was 1; events were not blocked by STI or by MOV. | 

SS; and the “interrupt-window exiting” VM-execution control was 1. 

8 NMI window. At the beginning of an instruction, there was no virtual-NMI blocking; events were not blocked by MOV 
SS; and the “NMI-window exiting” VM-execution control was 1. 

9 Task switch. Guest software attempted a task switch. 

10 CPUID. Guest software attempted to execute CPUID. 

11 GETSEC. Guest software attempted to execute GETSEC. 

12 HLT. Guest software attempted to execute HLT and the “HLT exiting” VM-execution control was 1. 

13 INVD. Guest software attempted to execute INVD. 

14 INVLPG. Guest software attempted to execute INVLPG and the “INVLPG exiting” VM-execution control was 1. 

15 RDPMC. Guest software attempted to execute RDPMC and the “RDPMC exiting” VM-execution control was 1. 

16 RDTSC. Guest software attempted to execute RDTSC and the “RDTSC exiting” VM-execution control was 1. 

17 RSM. Guest software attempted to execute RSM in SMM. 

18 VMCALL. VMCALL was executed either by guest software (causing an ordinary VM exit) or by the executive monitor 
(causing an SMM VM exit; see Section 30.15.2). 

19 VMCLEAR. Guest software attempted to execute VMCLEAR. 

20 VMLAUNCH. Guest software attempted to execute VMLAUNCH. 

21 VMPTRLD. Guest software attempted to execute VMPTRLD. 

22 VMPTRST. Guest software attempted to execute VMPTRST. 
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Table C-1. Basic Exit Reasons (Contd.) 


Basic Exit 
Reason Description 
23 VMREAD. Guest software attempted to execute VMREAD. 
24 VMRESUME. Guest software attempted to execute VMRESUME. 
25 VMWRITE. Guest software attempted to execute VMWRITE. 
26 VMXOFF. Guest software attempted to execute VMXOFF. 
27 VMXON. Guest software attempted to execute VMXON. 
28 Control-register accesses. Guest software attempted to access CRO, CR3, CR4, or CR8 using CLTS, LMSW, or 
MOV CR and the VM-execution control fields indicate that a VM exit should occur (see Section 24.1 for details). This 
basic exit reason is not used for trap-like VM exits following executions of the MOV to CR8 instruction when the “use 
TPR shadow” VM-execution control is 1. Such VM exits instead use basic exit reason 43. 
29 MOV DR. Guest software attempted a MOV to or from a debug register and the “MOV-DR exiting” VM-execution 
control was 1. 
30 1/0 instruction. Guest software attempted to execute an I/O instruction and either: 
1: The “use I/O bitmaps” VM-execution control was 0 and the “unconditional I/O exiting” VM-execution control was 1. 
2: The “use I/O bitmaps” VM-execution control was 1 and a bit in the I/O bitmap associated with one of the ports 
accessed by the I/O instruction was 1. 
31 RDMSR. Guest software attempted to execute RDMSR and either: 
1: The “use MSR bitmaps” VM-execution control was 0. 
2: The value of RCX is neither in the range QOOOOO000H - 00001 FFFH nor in the range COOOOO00H - CO001FFFH. 
3: The value of RCX was in the range OOOOO000H - OO001FFFH and the n* bit in read bitmap for low MSRs is 1, 
where n was the value of RCX. 
4: The value of RCX is in the range COOOOO00H - CO001FFFH and the n* bit in read bitmap for high MSRs is 1, where 
nis the value of RCX & 00001FFFH. 
32 WRMSR. Guest software attempted to execute WRMSR and either: 
1: The “use MSR bitmaps” VM-execution control was 0. 
2: The value of RCX is neither in the range QOOOOO000H - 00001 FFFH nor in the range COOOOO00H - CO001FFFH. 
3: The value of RCX was in the range OOOOOO00H - OO001FFFH and the nt” pit in write bitmap for low MSRs is 1, 
where n was the value of RCX. 
4: The value of RCX is in the range COOOO000H - COO01FFFH and the nt” bit in write bitmap for high MSRs is 1, 
where nis the value of RCX & 00001FFFH. 
33 VM-entry failure due to invalid guest state. A VM entry failed one of the checks identified in Section 25.3.1. 
34 VM-entry failure due to MSR loading. A VM entry failed in an attempt to load MSRs. See Section 25.4. 
36 MWAIT. Guest software attempted to execute MWAIT and the “MWAIT exiting” VM-execution control was 1. 
37 Monitor trap flag. A VM exit occurred due to the 1-setting of the “monitor trap flag” VM-execution control (see 
Section 24.5.2) or VM entry injected a pending MTF VM exit as part of VM entry (see Section 25.6.2). 
39 MONITOR. Guest software attempted to execute MONITOR and the “MONITOR exiting” VM-execution control was 1. 
40 PAUSE. Either guest software attempted to execute PAUSE and the “PAUSE exiting” VM-execution control was 1 or 
the “PAUSE-loop exiting” VM-execution control was 1 and quest software executed a PAUSE loop with execution 
time exceeding PLE_Window (see Section 24.1.3). 
41 VM-entry failure due to machine-check event. A machine-check event occurred during VM entry (see Section 
25.9). 
43 TPR below threshold. The logical processor determined that the value of bits 7:4 of the byte at offset O80H on the 
virtual-APIC page was below that of the TPR threshold VM-execution control field while the “use TPR shadow” V/M- 
execution control was 1 either as part of TPR virtualization (Section 28.1.2) or VM entry (Section 25.7.7). 
44 APIC access. Guest software attempted to access memory at a physical address on the APIC-access page and the 
“virtualize APIC accesses” VM-execution control was 1 (see Section 28.4). 
45 Virtualized EOI. EO! virtualization was performed for a virtual interrupt whose vector indexed a bit set in the EOI- 
exit bitmap. 
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Table C-1. Basic Exit Reasons (Contd.) 


Basic Exit 

Reason Description 

46 Access to GDTR or IDTR. Guest software attempted to execute LGDT, LIDT, SGDT, or SIDT and the “descriptor-table 
exiting” VM-execution control was 1. 

47 Access to LDTR or TR. Guest software attempted to execute LLDT, LTR, SLDT, or STR and the “descriptor-table 
exiting” VM-execution control was 1. 

48 EPT violation. An attempt to access memory with a guest-physical address was disallowed by the configuration of 
the EPT paging structures. 

49 EPT misconfiguration. An attempt to access memory with a guest-physical address encountered a misconfigured 
EPT paging-structure entry. 

50 INVEPT. Guest software attempted to execute INVEPT. 

51 RDTSCP. Guest software attempted to execute RDTSCP and the “enable RDTSCP” and “RDTSC exiting” VM-execution 
controls were both 1. 

52 VMX-preemption timer expired. The preemption timer counted down to zero. 

53 INVVPID. Guest software attempted to execute INVVPID. 

54 WBINVD or WBNOINVD. Guest software attempted to execute WBINVD or WBNOINVD and the “WBINVD exiting” 
\V/M-execution control was 1. 

55 XSETBV. Guest software attempted to execute XSETBV. 

56 APIC write. Guest software completed a write to the virtual-APIC page that must be virtualized by VMM software 
(see Section 28.4.3.3). 

57 RDRAND. Guest software attempted to execute RDRAND and the “RDRAND exiting” VM-execution control was 1. 

58 INVPCID. Guest software attempted to execute INVPCID and the “enable INVPCID” and “INVLPG exiting” 
\V/M-execution controls were both 1. 

59 VMFUNC. Guest software invoked a VM function with the VMFUNC instruction and the VM function either was not 
enabled or generated a function-specific condition causing a VM exit. 

60 ENCLS. Guest software attempted to execute ENCLS and “enable ENCLS exiting” VM-execution control was 1 and 
either (1) EAX < 63 and the corresponding bit in the ENCLS-exiting bitmap is 1; or (2) EAX 2 63 and bit 63 in the 
ENCLS-exiting bitmap is 1. 

61 RDSEED. Guest software attempted to execute RDSEED and the “RDSEED exiting” VM-execution control was 1. 

62 Page-modification log full. The processor attempted to create a page-modification log entry and the value of the 
PML index was not in the range 0-511. 

63 XSAVES. Guest software attempted to execute XSAVES, the “enable XSAVES/XRSTORS” was 1, and a bit was set in 
the logical-AND of the following three values: EDX:EAX, the [A32_XSS MSR, and the XSS-exiting bitmap. 

64 XRSTORS. Guest software attempted to execute XRSTORS, the “enable XSAVES/XRSTORS’ was 1, and a bit was set 
in the logical-AND of the following three values: EDX:EAX, the [A32_XSS MSR, and the XSS-exiting bitmap. 

66 SPP-related event. The processor attempted to determine an access's sub-page write permission and encountered 
an SPP miss or an SPP misconfiguration. See Section 27.2.4.2. 

67 UMWAIT. Guest software attempted to execute UMWAIT and the “enable user wait and pause” and “RDTSC exiting” 
\V/M-execution controls were both 1. 

68 TPAUSE. Guest software attempted to execute TPAUSE and the “enable user wait and pause” and “RDTSC exiting” 
\V/M-execution controls were both 1. 

69 LOADIWKEY. Guest software attempted to execute LOADIWKEY and the “LOADIWKEY exiting” VM-execution control 


was 1. 
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debug registers, 2-7 
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exception handling, 6-19 
external interrupts, 10-32 
fast system calls, 5-22 
GDTR register, 2-12, 2-13 
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IDTR register, 2-12 
initialization process, 2-8, 9-11 
interrupt and trap gates, 6-19 
interrupt controller, 10-32 
interrupt descriptors, 2-5 
interrupt handling, 6-19 
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processor, exceptions and interrupts, 19-6 
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82489DX, 21-27, 21-28 
Local APIC and I/O APICs, 10-4 
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description of, 6-5 
restarting a program or task after, 6-6 
AC (alignment check) flag, EFLAGS register, 2-11, 6-49, 21-6 
Access rights 
checking, 2-24 


checking caller privileges, 5-26 
description of, 5-24 
invalid values, 21-19 
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ADD instruction, 8-3 
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space, of task, 7-16 
Address translation 
in real-address mode, 19-2 
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overview, 3-6 
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Advanced power management 
C-state and Sub C-state, 14-27 
MWAIT extensions, 14-27 
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Alignment 
check exception, 2-11, 6-49, 21-11, 21-21 
checking, 5-27 
AM (alignment mask) flag 
CRO control register, 2-15, 21-18 
AND instruction, 8-3 
APIC, 10-41, 10-42 
APIC bus 
arbitration mechanism and protocol, 10-27, 10-34 
bus message format, 10-35, 10-48 
diagram of, 10-2, 10-3 
EOI message format, 10-15, 10-48 
nonfocused lowest priority message, 10-50 
short message format, 10-49 
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status cycles, 10-51 
structure of, 10-4 
See also 
local APIC 
APIC flag, CPUID instruction, 10-8 
APICID, 10-41, 10-45, 10-47 
APIC (see I/O APIC or Local APIC) 
ARPL instruction, 2-24, 5-27 
not supported in 64-bit mode, 2-24 
Atomic operations 
automatic bus locking, 8-3 


effects of a locked operation on internal processor caches, 8-5 


guaranteed, description of, 8-2 

overview of, 8-1, 8-3 

software-controlled bus locking, 8-3 
At-retirement 

counting, 18-98, 18-114 

events, 18-98, 18-103, 18-104, 18-114, 18-118 
Auto HALT restart 

field, SMM, 30-14 

SMM, 30-14 
Automatic bus locking, 8-3 
Automatic thermal monitoring mechanism, 14-28 
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B (busy) flag 
TSS descriptor, 7-5, 7-11, 7-16, 8-3 
B (default stack size) flag 
segment descriptor, 20-1, 21-33 
BO-B3 (BP condition detected) flags 
DR6 register, 17-3 
Backlink (see Previous task link) 
Base address fields, segment descriptor, 3-10 


BD (debug register access detected) flag, DR6 register, 17-3, 17-10 
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Binary numbers, 1-8 Cc 

BINIT# signal, 2-25 C (conforming) flag, segment descriptor, 5-11 
BIOS role in microcode updates, 9-38 C1 flag, x87 FPU status word, 21-8, 21-14 
Bit order, 1-7 — C2 flag, x87 FPU status word, 21-8 

BOUND instruction, 2-5, 6-4, 6-30 Cache control, 11-20 


BOUND range exceeded exception (#BR), 6-30 
BPO#, BP1#, BP2#, and BP3# pins, 17-38, 17-40 
Branch record 

branch trace message, 17-13 


adaptive mode, L1 Data Cache, 11-18 

cache management instructions, 11-17, 11-18 
cache mechanisms in IA-32 processors, 21-30 
caching terminology, 11-5 


IA-32e mode, 17-21 CD flag, CRO control register, 11-10, 21-19 
saving, 17-16, 17-25, 17-26, 17-35 choosing a memory type, 11-8 
saving as a branch trace message, 17-14 CPUID feature flag, 11-18 
structure, 17-35 flags and fields, 11-10 
structure of in BTS buffer, 17-20 flushing TLBs, 11-19 
Branch trace message (see BTM) G (global) flag 
Branch trace store (see BTS) page-directory entries, 11-13 
Breakpoint exception (#BP), 6-4, 6-28, 17-10 page-table entries, 11-13 


Breakpoints 


. internal caches, 11-1 
data breakpoint, 17-5 


MemTypeGet() function, 11-29 


data breakpoint exception conditions, 17-9 MemTypeSet() function, 11-31 
description of, 17-1 MESI protocol, 11-5, 11-9 
DRO-DR3 debug registers, 17-3 methods of caching available, 11-6 
example, 17-5 MTRR initialization, 11-29 
exception, 6-28 MTRR precedences, 11-28 

field recognition, 17-5, 17-6 MTRRs, description of, 11-20 


general-detect exception condition, 17-10 
instruction breakpoint, 17-5 

instruction breakpoint exception condition, 17-8 
I/O breakpoint exception conditions, 17-9 


multiple-processor considerations, 11-32 
NW flag, CRO control register, 11-13, 21-19 
operating modes, 11-12 

overview of, 11-1 


LENO - LEN3 (Length) fields page attribute table (PAT), 11-33 
DR7 register, 17-5 PCD flag 
R/WO-R/W3 (read/write) fields CR3 control register, 11-13 


_ DR7 register, 17-4 page-directory entries, 11-13, 11-33 
single-step exception condition, 17-10 page-table entries, 11-13, 11-33 
task-switch exception condition, 17-10 PGE (page global enable) flag, CR4 control register, 11-13 
BS (single step) flag, DR6 register, 17-3 precedence of controls, 11-13 
BSP flag, |A32_APIC_BASE MSR, 10-8 preventing caching, 11-16 
BSWAP instruction, 21-4 protocol, 11-9 
BT (task switch) flag, DR6 register, 17-3, 17-10 PWT flag 
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ale: : 
page-directory entries, 11-33 

DEBUGCTLMSR MSR, 17-40 page-table entries, 11-33 
BIMs (branch trace messages) remapping memory types, 11-29 

description of, 17-13 setting up memory ranges with MTRRs, 11-22 


enabling, 17-12, 17-23, 17-24, 17-34, 17-37, 17-38 shared mode, L1 Data Cache, 11-18 

TR (trace message enable) flag iable- : MTRR 11-23 41-25 
MSR_DEBUGCTLA MSR, 17-34 Gade 
MSR_DEBUGCTLB MSR, 17-12, 17-37, 17-38 cache hit, 11-5 


BTR instruction, 8-3 cache line, 11-5 


BTS buffer cache line fill, 11-5 
description of, 17-18 cache write hit, 11-5 
introduction to, 17-11, 17-14 description of, 11-1 


records in, 17-20 effects of a locked operation on internal processor caches, 8-5 
setting up, 17-23 enabling, 9-7 
structure of, 17-19, 17-21, 18-22 management, instructions, 2-24, 11-17 
BTS instruction, 8-3 Caching 
BTS (branch trace store) facilities cache control protocol, 11-9 
availability of, 17-33 cacheline, 11-5 
BTS_UNAVAILABLE flag, cache management instructions, 11-17 
IA32_MISC_ENABLE MSR, 17-18 cache mechanisms in IA-32 processors, 21-30 


introduction to, 17-11 : F 
: , caching terminology, 11-5 
setting up BTS buffer, 17-23 choosing a memory type, 11-8 


writing an interrupt service routine for, 17-24 flushing TLBs, 11-19 
BTS_UNAVAILABLE, 17-18 implicit caching, 11-19 
Built-in self-test (BIST) internal caches, 11-1 
description of, 9-1 L1 (level 1) cache, 11-4 
performing, 9-5 L2 (level 2) cache, 11-4 
Bus : L3 (level 3) cache, 11-4 
aight with MCA, 15-27 methods of caching available, 11-6 


: MTRRs, description of, 11-20 
locking, 8-3, 21-35 operating modes, 11-12 
Byte order, 1-7 overview of, 11-1 
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self-modifying code, effect on, 11-18, 21-30 
snooping, 11-6 
store buffer, 11-20 
TLBs, 11-5 
UC (strong uncacheable) memory type, 11-6 
UC- (uncacheable) memory type, 11-6 
WB (write back) memory type, 11-7 
WC (write combining) memory type, 11-7 
WP (write protected) memory type, 11-7 
write-back caching, 11-6 
WT (write through) memory type, 11-7 
Call gates 
16-bit, interlevel return from, 21-33 
accessing a code segment through, 5-15 
description of, 5-13 
for 16-bit and 32-bit code modules, 20-1 
IA-32e mode, 5-14 
introduction to, 2-4 
mechanism, 5-15 
privilege level checking rules, 5-16 
CALL instruction, 2-5, 3-8, 5-10, 5-15, 5-20, 7-2, 7-9, 7-11, 20-5 
Caller access privileges, checking, 5-26 
Calls 
16 and 32-bit code segments, 20-3 
controlling operand-size attribute, 20-5 
returning from, 5-20 
Catastrophic shutdown detector 
Thermal monitoring 
catastrophic shutdown detector, 14-29 
catastrophic shutdown detector, 14-28 
CCO and CC1 (counter control) fields, CESR MSR (Pentium processor), 
18-134 
CD (cache disable) flag, CRO control register, 2-15, 9-7, 11-10, 11-12, 
11-13, 11-16, 11-32, 21-18, 21-19, 21-30 


CESR (control and event select) MSR (Pentium processor), 18-133, 18-134 


CLFLSH feature flag, CPUID instruction, 9-8 
CLFLUSH instruction, 2-15, 9-8, 11-17 
CLI instruction, 6-7 
Clocks 
counting processor clocks, 18-135 
Hyper-Threading Technology, 18-135 
nominal CPI, 18-135 
non-halted clockticks, 18-135 
non-halted CPI, 18-135 
non-sleep Clockticks, 18-135 
time stamp counter, 18-135 
CLTS instruction, 2-23, 5-24, 24-2, 24-7 
Cluster model, local APIC, 10-25 
CMOVcc instructions, 21-4 
CMPXCHG instruction, 8-3, 21-4 
CMPXCHG8B instruction, 8-3, 21-5 
Code modules 
16 bit vs. 32 bit, 20-1 
mixing 16-bit and 32-bit code, 20-1 
sharing data, mixed-size code segs, 20-3 
transferring control, mixed-size code segs, 20-3 
Code segments 
accessing data in, 5-9 
accessing through a call gate, 5-15 
description of, 3-12 
descriptor format, 5-2 
descriptor layout, 5-2 
direct calls or jumps to, 5-10 
paging of, 2-6 
pointer size, 20-4 
privilege level checks 
transferring control between code segs, 5-10 
Compatibility 
IA-32 architecture, 21-1 
software, 1-7 
Compatibility mode 
code segment descriptor, 5-3 
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code segment descriptors, 9-12 

control registers, 2-13 

CS.L and CS.D, 9-12 

debug registers, 2-24 

EFLAGS register, 2-11 

exception handling, 2-5 

gates, 2-4 

GDTR register, 2-12, 2-13 

global and local descriptor tables, 2-4 

IDTR register, 2-12 

interrupt handling, 2-5 

L flag, 3-12, 5-4 

memory management, 2-6 

operation, 9-12 

segment loading instructions, 3-9 

segments, 3-5 

switching to, 9-12 

SYSCALL and SYSRET, 5-22 

SYSENTER and SYSEXIT, 5-21 

system flags, 2-11 

system registers, 2-7 

task register, 2-13 

See also: 64-bit mode, |A-32e mode 
Condition code flags, x87 FPU status word 

compatibility information, 21-8 
Conforming code segments 

accessing, 5-12 

C (conforming) flag, 5-11 

description of, 3-13 
Context, task (see Task state) 
Control registers 

64-bit mode, 2-13 


CRO, 2-13 

CR1 (reserved), 2-13 
CR2, 2-13 

CR3 (PDBR), 2-6, 2-13 
CR4, 2-13 


description of, 2-13 
introduction to, 2-6 
Coprocessor segment 
overrun exception, 6-35, 21-12 
Counter mask field 
Perf€vtSel0 and Perf€vtSel1 MSRs (P6 family processors), 18-5, 
18-132 
CPL 
description of, 5-7 
field, CS segment selector, 5-2 
CPUID instruction 
availability, 21-5 
control register flags, 2-20 
detecting features, 21-2 
serializing instructions, 8-17 
syntax for data, 1-9 
CRO control register, 21-8 
description of, 2-13 
introduction to, 2-6 
state following processor reset, 9-2 
CR1 control register (reserved), 2-13 
CR2 control register 
description of, 2-13 
introduction to, 2-6 
CR3 control register (PDBR) 
associated with a task, 7-1, 7-3 
description of, 2-13 
in TSS, 7-4, 7-17 
introduction to, 2-6 
loading during initialization, 9-10 
memory management, 2-6 
page directory base address, 2-6 
page table base address, 2-5 
CR4 control register 
description of, 2-13 
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enabling control functions, 21-2 
inclusion in |A-32 architecture, 21-18 
introduction to, 2-6 
VMX usage of, 22-3 
CR8 register, 2-7 
64-bit mode, 2-13 
compatibility mode, 2-13 
description of, 2-13 
task priority level bits, 2-20 
when available, 2-13 
CS register, 21-11 
state following initialization, 9-5 
C-state, 14-27 
CTRO and CTR1 (performance counters) MSRs (Pentium processor), 
18-133, 18-135 
Current privilege level (see CPL) 
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D (default operation size) flag 
segment descriptor, 20-1, 21-33 
Data breakpoint exception conditions, 17-9 
Data segments 
description of, 3-12 
descriptor layout, 5-2 
expand-down type, 3-11 
paging of, 2-6 
privilege level checking when accessing, 5-8 
DE (debugging extensions) flag, CR4 control register, 2-17, 21-18, 21-20 
Debug exception (#DB), 6-7, 6-25, 7-5, 17-7, 17-13, 17-41 
Debug store (see DS) 
DEBUGCTLMSR MSR, 17-39, 17-40 
Debugging facilities 
breakpoint exception (#BP), 17-1 
debug exception (#DB), 17-1 
DR6 debug status register, 17-1 
DR7 debug control register, 17-1 
exceptions, 17-6 
INT3 instruction, 17-1 
last branch, interrupt, and exception recording, 17-1, 17-11 
masking debug exceptions, 6-7 
overview of, 17-1 
performance-monitoring counters, 18-1 
registers 
description of, 17-2 
introduction to, 2-6 
loading, 2-24 
RF (resume) flag, EFLAGS, 17-1 
see DS (debug store) mechanism 
T (debug trap) flag, TSS, 17-1 
TF (trap) flag, EFLAGS, 17-1 
DEC instruction, 8-3 
Denormal operand exception (#D), 21-10 
Denormalized operand, 21-12 
Device-not-available exception (HNM), 2-15, 2-23, 6-32, 9-7, 21-11, 
21-12 
DFR 
Destination Format Register, 10-39, 10-42, 10-47 
Digital readout bits, 14-36, 14-39 
DIV instruction, 6-24 
Divide configuration register, local APIC, 10-16 
Divide-error exception (HDE), 6-24, 21-21 
Double-fault exception (#DF), 6-33, 21-27 
DPL (descriptor privilege level) field, segment descriptor, 3-11, 5-2, 5-4, 
DRO-DR3 breakpoint-address registers, 17-1, 17-3, 17-38, 17-40, 17-41 
DR4-DR5 debug registers, 17-3, 21-20 
DR6 debug status register, 17-3 
BO-B3 (BP detected) flags, 17-3 
BD (debug register access detected) flag, 17-3 
BS (single step) flag, 17-3 
BT (task switch) flag, 17-3 


INDEX-4 Vol. 3D 


debug exception (#DB), 6-25 
reserved bits, 21-20 
DR7 debug control register, 17-4 
GO-G3 (global breakpoint enable) flags, 17-4 
GD (general detect enable) flag, 17-4 
GE (global exact breakpoint enable) flag, 17-4 
LO-L3 (local breakpoint enable) flags, 17-4 
LE local exact breakpoint enable) flag, 17-4 
LENO-LEN3 (Length) fields, 17-4 
R/WO-R/W3 (read/write) fields, 17-4, 21-20 
DS feature flag, CPUID instruction, 17-18, 17-33, 17-37, 17-38 
DS save area, 17-19, 17-20, 17-21 
DS (debug store) mechanism 
availability of, 18-108 
description of, 18-108 
DS feature flag, CPUID instruction, 18-108 
DS save area, 17-18, 17-20 
IA-32e mode, 17-20 
interrupt service routine (DS ISR), 17-24 
setting up, 17-22 
Dual-core technology 
architecture, 8-32 
logical processors supported, 8-25 
MTRR memory map, 8-33 
multi-threading feature flag, 8-24 
performance monitoring, 18-122 
specific features, 21-4 
Dual-monitor treatment, 30-19 
D/B (default operation size/default stack pointer size and/or upper bound) 
flag, segment descriptor, 3-11, 5-4 


E (edge detect) flag 
Perf€vtSel0 and PerfEvtSell MSRs (P6 family), 18-4 
E (edge detect) flag, Perf€vtSel0 and Perf€vtSel1 MSRs (P6 family 
processors), 18-131 
E (expansion direction) flag 
segment descriptor, 5-2, 5-4 
€ (MTRRs enabled) flag 
|A32_MTRR_DEF_TYPE MSR, 11-23 
EFLAGS register 
identifying 32-bit processors, 21-6 
introduction to, 2-6 
new flags, 21-6 
saved in TSS, 7-4 
system flags, 2-9 
EIP register, 21-11 
saved in TSS, 7-5 
state following initialization, 9-5 
EM (emulation) flag 
CRO control register, 2-15, 2-16, 6-32, 9-6, 9-7, 12-1, 13-3 
EMMS instruction, 12-3 
Enhanced Intel SpeedStep Technology 
ACPI 3.0 specification, 14-1 
IA32_APERF MSR, 14-2 
|A32_MPERF MSR, 14-2 
|A32_PERF_CTL MSR, 14-1 
IA32_PERF_STATUS MSR, 14-1 
introduction, 14-1 
multiple processor cores, 14-1 
performance transitions, 14-1 
P-state coordination, 14-1 
See also: thermal monitoring 
EOI 
End Of Interrupt register, 10-39 
Error code, 16-3, 16-7, 16-10, 16-13, 16-15 
architectural MCA, 16-1, 16-3, 16-7, 16-10, 16-13, 16-15 
decoding IA32_MCi_STATUS, 16-1, 16-3, 16-7, 16-10, 16-13, 16-15 
exception, description of, 6-17 
external bus, 16-1, 16-3, 16-7, 16-10, 16-13, 16-15 
memory hierarchy, 16-3, 16-7, 16-10, 16-13, 16-15 


pushing on stack, 21-32 
watchdog timer, 16-1, 16-3, 16-7, 16-10, 16-13, 16-15 
Error numbers 
VM-instruction error field, 29-31 
Error signals, 21-11 
Error-reporting bank registers, 15-2 
ERROR# 
input, 21-16 
output, 21-16 
ESO and ES1 (event select) fields, CESR MSR (Pentium processor), 18-134 
ESR 
Error Status Register, 10-40 
ET (extension type) flag, CRO control register, 2-15, 21-8 
Event select field, Perf€vtSel0 and Perf€vtSell MSRs (P6 family 
processors), 18-4, 18-95, 18-131 
Events 
at-retirement, 18-114 
at-retirement (Pentium 4 processor), 18-103 
non-retirement (Pentium 4 processor), 18-103 
Exception handler 
calling, 6-11 
defined, 6-1 
flag usage by handler procedure, 6-17 
machine-check exception handler, 15-28 
machine-check exceptions (#MC), 15-28 
machine-error logging utility, 15-28 
procedures, 6-12 
protection of handler procedures, 6-16 
task, 6-17, 7-2 
Exceptions 
alignment check, 21-11 
classifications, 6-5 
compound error codes, 15-21 
conditions checked during a task switch, 7-13 
coprocessor segment overrun, 21-12 
description of, 2-5, 6-1 
device not available, 21-12 
double fault, 6-33 
error code, 6-17 
execute-disable bit, 5-32 
floating-point error, 21-12 
general protection, 21-12 
handler mechanism, 6-12 
handler procedures, 6-12 
handling, 6-11 
handling in real-address mode, 19-4 
handling in SMM, 30-11 
handling in virtual-8086 mode, 19-11 
handling through a task gate in virtual-8086 mode, 19-14 
handling through a trap or interrupt gate in virtual-8086 mode, 19-12 
IA-32e mode, 2-5 
IDT, 6-9 
initializing for protected-mode operation, 9-10 
invalid-opcode, 21-5 
masking debug exceptions, 6-7 
masking when switching stack segments, 6-8 
MCA error codes, 15-20 
MMxX instructions, 12-1 
notation, 1-9 
overview of, 6-1 
priority of, 21-22 
priority of, x87 FPU exceptions, 21-10 
reference information on all exceptions, 6-23 
reference information, 64-bit mode, 6-19 
restarting a task or program, 6-5 
segment not present, 21-12 
simple error codes, 15-21 
sources of, 6-4 
summary of, 6-2 
vectors, 6-1 
Executable, 3-11 
Execute-disable bit capability 
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conditions for, 5-30 
CPUID flag, 5-30 
detecting and enabling, 5-30 
exception handling, 5-32 
page-fault exceptions, 6-44 
protection matrix for |A-32e mode, 5-31 
protection matrix for legacy modes, 5-31 
reserved bit checking, 5-31 
Exit-reason numbers 
VM entries & exits, C-1 
Expand-down data segment type, 3-11 
Extended signature table, 9-31 
extended signature table, 9-31 
External bus errors, detected with machine-check architecture, 15-27 
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F2XM1 instruction, 21-13 
Family O6H, 16-1 
Family OFH, 16-1 
microcode update facilities, 9-28 
Faults 
description of, 6-5 
restarting a program or task after, 6-5 
FCMOVcc instructions, 21-4 
FCOMI instruction, 21-4 
FCOMIP instruction, 21-4 
FCOS instruction, 21-13 
FDIV instruction, 21-11, 21-12 
FE (fixed MTRRs enabled) flag, IA32_MTRR_DEF_TYPE MSR, 11-23 
Feature 
determination, of processor, 21-2 
information, processor, 21-2 
FINIT/FNINIT instructions, 21-8, 21-16 
FIX (fixed range registers supported) flag, IA32_MTRRCAPMSR, 11-22 
Fixed-range MTRRs 
description of, 11-23 
Flat segmentation model, 3-3 
FLD instruction, 21-14 
FLDENV instruction, 21-12 
FLDL2E instruction, 21-14 
FLDL2T instruction, 21-14 
FLDLG2 instruction, 21-14 
FLDLN2 instruction, 21-14 
FLDPI instruction, 21-14 
Floating-point error exception (#MF), 21-12 
Floating-point exceptions 
denormal operand exception (#D), 21-10 
invalid operation (#1), 21-14 
numeric overflow (#0), 21-10 
numeric underflow (#U), 21-10 
saved CS and EIP values, 21-11 
FLUSH# pin, 6-3 
FNSAVE instruction, 12-4 
Focus processor, local APIC, 10-27 
FORCEPR# log, 14-35, 14-38 
FORCPR# interrupt enable bit, 14-36 
FPATAN instruction, 21-13 
FPREM instruction, 21-8, 21-11, 21-12, 21-13 
FPREM1 instruction, 21-8, 21-13 
FPTAN instruction, 21-8, 21-13 
FRSTOR instruction, 12-4, 21-11, 21-12 
FSAVE instruction, 12-3, 12-4 
FSAVE/FNSAVE instructions, 21-11, 21-14 
FSCALE instruction, 21-12 
FSIN instruction, 21-13 
FSINCOS instruction, 21-13 
FSQRT instruction, 21-11, 21-12 
FSTENV instruction, 12-3 
FSTENV/FNSTENV instructions, 21-14 
FTAN instruction, 21-8 
FUCOM instruction, 21-13 
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FUCOMI instruction, 21-4 architecture description, 8-27 
FUCOMIP instruction, 21-4 caches, 8-31 
FUCOMP instruction, 21-13 debug registers, 8-30 
FUCOMPP instruction, 21-13 description of, 8-24, 21-3, 21-4 
FWAIT instruction, 6-32 detecting, 8-36, 8-37, 8-42, 8-43 
FXAM instruction, 21-14 executing multiple threads, 8-26 
FXRSTOR instruction, 2-18, 2-19, 9-8, 12-3, 12-4, 13-2, 13-6 execution-based timing loops, 8-55 
FXSAVE instruction, 2-18, 2-19, 9-8, 12-3, 12-4, 13-2, 13-6 external signal compatibility, 8-32 
FXSR feature flag, CPUID instruction, 9-8 halting logical processors, 8-53 
FXTRACT instruction, 21-10, 21-14 handling interrupts, 8-26 
HLT instruction, 8-49 
G 1A32_MISC_ENABLE MSR, 8-30, 8-33 
initializing |[A-32 processors with, 8-25 
G (global) flag introduction of into the IA-32 architecture, 21-3, 21-4 
page-directory entries, 11-13 local a, 8-28 
page-table entries, 11-13 local APIC 
G (granularity) flag functionality in logical processor, 8-29 
segment descriptor, 3-10, 3-11, 5-2, 5-4 logical processors, identifying, 8-38 
GO-G3 (global breakpoint enable) flags machine check architecture, 8-29 
DR7 register, 17-4 managing idle and blocked conditions, 8-49 
Gate descriptors mapping resources, 8-34 
call gates, 5-13 memory ordering, 8-30 
description of, 5-13 microcode update resources, 8-30, 8-33, 9-35 
IA-32e mode, 5-14 MP systems, 8-27 
Gates, 2-4 MTRRs, 8-29, 8-33 
IA-32e mode, 2-4 multi-threading feature flag, 8-24 
GD (general detect enable) flag multi-threading support, 8-24 
DR7 register, 17-4, 17-10 PAT, 8-29 
GDT PAUSE instruction, 8-49, 8-50 
description of, 2-3, 3-15 performance monitoring, 18-117, 18-122 
|A-32e mode, 2-4 performance monitoring counters, 8-30, 8-33 
index field of segment selector, 3-7 placement of locks and semaphores, 8-55 
initializing, 9-10 required operating system support, 8-51 
paging of, 2-6 scheduling multiple threads, 8-54 
pointers to exception/interrupt handlers, 6-12 self modifying code, 8-31 
segment descriptors in, 3-9 serializing instructions, 8-30 
selecting with TI flag of segment selector, 3-7 spin-wait loops 
task switching, 7-9 PAUSE instructions in, 8-52, 8-54 
task-gate descriptor, 7-8 thermal monitor, 8-32 
TSS descriptors, 7-5 TLBs, 8-31 
use in address translation, 3-6 
GDTR register 
description of, 2-3, 2-6, 2-12, 3-15 I 
IA-32e mode, 2-4, 2-12 IA-32 Intel architecture 
limit, 5-5 compatibility, 21-1 
loading during initialization, 9-10 processors, 21-1 
storing, 3-15 IA32e mode 
GE (global exact breakpoint enable) flag registers and mode changes, 9-12 
DR7 register, 17-4, 17-9 IA-32e mode 
General-detect exception condition, 17-10 call gates, 5-14 
General-protection exception (#GP), 3-12, 5-6, 5-7, 5-11, 5-12, 6-9, code segment descriptor, 5-3 
6-16, 6-41, 7-5, 17-3, 21-12, 21-21, 21-34, 21-35 D flag, 5-4 
General-purpose registers, saved in TSS, 7-4 data structures and initialization, 9-11 
Global control MSRs, 15-2 debug registers, 2-7 
Global descriptor table register (see GDTR) debug store area 
Global descriptor table (see GDT) descriptors, 2-4 
DPL field, 5-4 
H exceptions during initialization, 9-12 
feature-enable register, 2-7 
HALT state gates, 2-4 
relationship to SMI interrupt, 30-3, 30-14 global and local descriptor tables, 2-4 
Hardware reset IA32_EFER MSR, 2-7, 5-30 
description of, 9-1 initialization process, 9-11 
processor state after reset, 9-2 interrupt stack table, 6-22 
state of MTRRs following, 11-20 interrupts and exceptions, 2-5 
value of SMBASE following, 30-4 IRET instruction, 6-21 
Hexadecimal numbers, 1-8 L flag, 3-12, 5-4 
high-temperature interrupt enable bit, 14-36, 14-39 logical address, 3-7 
HITM# line, 11-6 MOV CRn, 9-11 
HLT instruction, 2-25, 5-24, 6-34, 24-2, 30-14 MTRR calculations, 11-27 
Hyper-Threading Technology NXE bit, 5-30 
architectural state of a logical processor, 8-33 page level protection, 5-30 
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paging, 2-6 
PDE tables, 5-31 
PDP tables, 5-31 
PML4 tables, 5-31 
PTE tables, 5-31 
registers and data structures, 2-1 
segment descriptor tables, 3-16, 5-3 
segment descriptors, 3-9 
segment loading instructions, 3-9 
segmentation, 3-5 
stack switching, 5-19, 6-21 
SYSCALL and SYSRET, 5-22 
SYSENTER and SYSEXIT, 5-21 
system descriptors, 3-14 
system registers, 2-7 
task switching, 7-19 
task-state segments, 2-5 
terminating mode operation, 9-12 
See also: 64-bit mode, compatibility mode 
|A32_APERF MSR, 14-2 
IA32_APIC_BASE MSR, 8-18, 8-20, 10-6, 10-8 
IA32_CLOCK_MODULATION MSR, 8-32, 14-7, 14-14, 14-15, 14-16, 
14-17, 14-18, 14-21, 14-22, 14-23, 14-24, 14-32, 14-33, 
14-34, 14-35, 14-43, 14-44, 14-45, 14-46, 14-47 
IA32_DEBUGCTL MSR, 26-27 
IA32_DS_AREA MSR, 17-18, 17-22, 18-101, 18-116 
IA32_EFER MSR, 2-7, 2-8, 5-30, 26-28 
IA32_FEATURE_CONTROL MSR, 22-3 
IA32_KernelGSbase MSR, 2-7 
IA32_LSTAR MSR, 2-7, 5-22 
IA32_MCG_CAP MSR, 15-2, 15-28 
IA32_MCG_CTL MSR, 15-2, 15-4 
IA32_MCG_EAX MSR, 15-12 
IA32_MCG_EBP MSR, 15-12 
IA32_MCG_EBX MSR, 15-12 
IA32_MCG_ECX MSR, 15-12 
IA32_MCG_EDI MSR, 15-12 
IA32_MCG_EDX MSR, 15-12 
IA32_MCG_EFLAGS MSR, 15-12 
IA32_MCG_EIP MSR, 15-12 
IA32_MCG_ESI MSR, 15-12 
IA32_MCG_ESP MSR, 15-12 
IA32_MCG_MISC MSR, 15-12, 15-13 
IA32_MCG_R10 MSR, 15-13 
IA32_MCG_R11 MSR, 15-13 
IA32_MCG_R12 MSR, 15-13 
IA32_MCG_R13 MSR, 15-13 
IA32_MCG_R14 MSR, 15-13 
IA32_MCG_R15 MSR, 15-13 
IA32_MCG_R8 MSR, 15-13 
IA32_MCG_R9 MSR, 15-13 
IA32_MCG_RAX MSR, 15-12 
IA32_MCG_RBP MSR, 15-12 
IA32_MCG_RBX MSR, 15-12 
IA32_MCG_RCX MSR, 15-12 
IA32_MCG_RDI MSR, 15-12 
IA32_MCG_RDX MSR, 15-12 
IA32_MCG_RESERVEDn MSR, 15-12 
IA32_MCG_RFLAGS MSR, 15-12 
IA32_MCG_RIP MSR, 15-12 
|A32_MCG_RSI MSR, 15-12 
IA32_MCG_RSP MSR, 15-12 
IA32_MCG_STATUS MSR, 15-2, 15-4, 15-28, 15-30, 26-3 
IA32_MCi_ADDR MSR, 15-9, 15-30 
IA32_MCi_CTL, 15-5 
IA32_MCi_CTL MSR, 15-5 
IA32_MCi_MISC MSR, 15-9, 15-11, 15-12, 15-30 
IA32_MCi_STATUS MSR, 15-6, 15-28, 15-30 
decoding for Family O6H, 16-1 
decoding for Family OFH, 16-1, 16-3, 16-7, 16-10, 16-13, 16-15 
IA32_MISC_ENABLE MSR, 14-1, 14-29, 17-18, 17-33, 18-101 
IA32_MPERF MSR, 14-1, 14-2 


IA32_MTRRCAP MSR, 11-21, 11-22 
IA32_MTRR_DEF_TYPE MSR, 11-22 
IA32_MTRR_FIXn, fixed ranger MTRRs, 11-23 
IA32_PAT_CR MSR, 11-34 
IA32_PEBS_ENABLE MSR, 18-99, 18-101, 18-116 
IA32_PERF_CTL MSR, 14-1 
IA32_PERF_STATUS MSR, 14-1 
IA32_STAR MSR, 5-22 
IA32_STAR_CS MSR, 2-7 
IA32_SYSCALL_FLAG_MASK MSR, 2-7 
IA32_SYSENTER_CS MSR, 5-21, 5-22, 26-22 
IA32_SYSENTER_EIP MSR, 5-21 
IA32_SYSENTER_ESP MSR, 5-21, 26-28 
IA32_THERM_INTERRUPT MSR, 14-31, 14-34, 14-36 
FORCPR# interrupt enable bit, 14-36 
high-temperature interrupt enable bit, 14-36, 14-39 
low-temperature interrupt enable bit, 14-36, 14-39 
overheat interrupt enable bit, 14-36, 14-39 
THERMTRIP# interrupt enable bit, 14-36, 14-39 
threshold #1 interrupt enable bit, 14-37, 14-39 
threshold #1 value, 14-36, 14-39 
threshold #2 interrupt enable, 14-37, 14-40 
threshold #2 value, 14-37, 14-39 
IA32_THERM_STATUS MSR, 14-34 
digital readout bits, 14-36, 14-39 
out-of-spec status bit, 14-35, 14-38 
out-of-spec status log, 14-35, 14-38, 14-39 
PROCHOT# or FORCEPR# event bit, 14-34, 14-38, 14-39 
PROCHOT# or FORCEPR# log, 14-35, 14-38 
resolution in degrees, 14-36 
thermal status bit, 14-34, 14-38 
thermal status log, 14-34, 14-38 
thermal threshold #1 log, 14-35, 14-38, 14-39 
thermal threshold #1 status, 14-35, 14-38 
thermal threshold #2 log, 14-35, 14-38 
thermal threshold #2 status, 14-35, 14-38, 14-39 
validation bit, 14-36 
IA32_VMX_BASIC MSR, 23-3, A-1, A-2 
IA32_VMX_CRO_FIXEDO MSR, A-6 
IA32_VMX_CRO_FIXED1 MSR, A-6 
IA32_VMX_CR4_FIXEDO MSR, A-7 
IA32_VMX_CR4_FIXED1 MSR, A-7 
IA32_VMX_ENTRY_CTLS MSR, A-2, A-5 
IA32_VMX_EXIT_CTLS MSR, A-2, A-4, A-5 
IA32_VMX_MISC MSR, 23-6, 25-3, 25-13, 30-26, A-6 
IA32_VMX_PINBASED_CTLS MSR, A-2, A-3 
IA32_VMX_PROCBASED_CTLS MSR, 23-10, A-2, A-3, A-4, A-8 
IA32_VMX_VMCS_ENUM MSR, A-7 
ICR 
Interrupt Command Register, 10-39, 10-42, 10-48 
ID (identification) flag 
EFLAGS register, 2-11, 21-6 
IDIV instruction, 6-24, 21-21 
IDT 
64-bit mode, 6-19 
call interrupt & exception-handlers from, 6-11 
change base & limit in real-address mode, 19-5 
description of, 6-9 
handling NMIs during initialization, 9-9 
initializing protected-mode operation, 9-10 
initializing real-address mode operation, 9-8 
introduction to, 2-5 
limit, 21-27 
paging of, 2-6 
structure in real-address mode, 19-5 
task switching, 7-10 
task-gate descriptor, 7-8 
types of descriptors allowed, 6-10 
use in real-address mode, 19-4 
IDTR register 
description of, 2-12, 6-9 
IA-32e mode, 2-12 
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introduction to, 2-5 
limit, 5-5 
loading in real-address mode, 19-5 
storing, 3-16 
IE (invalid operation exception) flag 
x87 FPU status word, 21-8 
IEEE Standard 754 for Binary Floating-Point Arithmetic, 21-9, 21-10, 
21-12, 21-13, 21-14 
IF (interrupt enable) flag 
EFLAGS register, 2-10, 2-11, 6-7, 6-10, 6-17, 19-4, 19-19, 30-11 
IN instruction, 8-15, 21-35, 24-2 
INC instruction, 8-3 
Index field, segment selector, 3-7 
INIT interrupt, 10-3 
Initial-count register, local APIC, 10-16, 10-17 
Initialization 
built-in self-test (BIST), 9-1, 9-5 
CS register state following, 9-5 
EIP register state following, 9-5 
example, 9-14 
first instruction executed, 9-5 
hardware reset, 9-1 
IA-32e mode, 9-11 
IDT, protected mode, 9-10 
IDT, real-address mode, 9-8 
Intel486 SX processor and Intel 487 SX math coprocessor, 21-16 
location of software-initialization code, 9-5 
machine-check initialization, 15-19 
model and stepping information, 9-5 
multitasking environment, 9-10, 9-11 
overview, 9-1 
paging, 9-10 
processor state after reset, 9-2 
protected mode, 9-9 
real-address mode, 9-8 
RESET# pin, 9-1 
setting up exception- and interrupt-handling facilities, 9-10 
x87 FPU, 9-5 
INIT# pin, 6-3, 9-1 
INIT# signal, 2-25, 22-4 
INS instruction, 17-9 
Instruction operands, 1-8 
Instruction-breakpoint exception condition, 17-8 
Instructions 
new instructions, 21-4 
obsolete instructions, 21-5 
privileged, 5-23 
serializing, 8-17, 8-30, 21-16 
supported in real-address mode, 19-3 
system, 2-7, 2-22 
INS/INSB/INSW/INSD instruction, 24-2 
INT 3 instruction, 2-5, 6-28, 17-8 
INT instruction, 2-5, 5-10 
INT n instruction, 3-8, 6-1, 6-4, 17-10 
INT (APIC interrupt enable) flag, Perf€vtSel0 and Perf€vtSel1 MSRs (P6 
family processors), 18-5, 18-132 
INT15 and microcode updates, 9-42 
INT3 instruction, 3-8, 6-4 
Intel 287 math coprocessor, 21-7 
Intel 387 math coprocessor system, 21-7 
Intel 487 SX math coprocessor, 21-7, 21-16 
Intel 8086 processor, 21-7 
Intel Core Solo and Intel Core Duo processors 
event mask (Umask), 18-93, 18-94 
last branch, interrupt, exception recording, 17-36 
notes on P-state transitions, 14-1 
performance monitoring, 18-93, 18-94 
sub-fields layouts, 18-93, 18-94 
time stamp counters, 17-41 
Intel NetBurst microarchitecture, 1-3 
Intel software network link, 1-10 
Intel SpeedStep Technology 
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Intel VTune Performance Analyzer 


related information, 1-10 


Intel Xeon processor, 1-1 


last branch, interrupt, and exception recording, 17-33 
time-stamp counter, 17-41 


Intel Xeon processor MP 


with 8MB L3 cache, 18-122, 18-124 


Intel286 processor, 21-7 

Intel386 DX processor, 21-7 
Intel386 SL processor, 2-7 
Intel486 DX processor, 21-7 
Intel486 SX processor, 21-7, 21-16 
Interprivilege level calls 


call mechanism, 5-15 
stack switching, 5-17 


Interprocessor interrupt (IPIs), 10-1 
Interprocessor interrupt (IPI) 


in MP systems, 10-1 


interrupt, 6-13 

Interrupt Command Register, 10-39 

Interrupt command register (ICR), local APIC, 10-20 
Interrupt gates 


16-bit, interlevel return from, 21-33 

clearing IF flag, 6-7, 6-17 

difference between interrupt and trap gates, 6-17 

for 16-bit and 32-bit code modules, 20-1 

handling a virtual-8086 mode interrupt or exception through, 19-12 
in IDT, 6-10 

introduction to, 2-4, 2-5 

layout of, 6-10 


Interrupt handler 


calling, 6-11 

defined, 6-1 

flag usage by handler procedure, 6-17 
procedures, 6-12 

protection of handler procedures, 6-16 
task, 6-17, 7-2 


Interrupts 


automatic bus locking, 21-35 

control transfers between 16- and 32-bit code modules, 20-6 
description of, 2-5, 6-1 

destination, 10-27 

distribution mechanism, local APIC, 10-26 
enabling and disabling, 6-6 

handling, 6-11 

handling in real-address mode, 19-4 

handling in SMM, 30-11 

handling in virtual-8086 mode, 19-11 

handling multiple NMIs, 6-6 

handling through a task gate in virtual-8086 mode, 19-14 
handling through a trap or interrupt gate in virtual-8086 mode, 19-12 
IA-32e mode, 2-5, 2-12 

IDT, 6-9 

IDTR, 2-12 

initializing for protected-mode operation, 9-10 
interrupt descriptor table register (see IDTR) 
interrupt descriptor table (see IDT) 

list of, 6-2, 19-6 

local APIC, 10-1 

maskable hardware interrupts, 2-10 

masking maskable hardware interrupts, 6-7 
masking when switching stack segments, 6-8 
message signalled interrupts, 10-35 

on-die sensors for, 14-28 

overview of, 6-1 

priority, 10-29 

propagation delay, 21-27 

real-address mode, 19-6 

restarting a task or program, 6-5 

software, 6-57 

sources of, 10-1 


summary of, 6-2 
thermal monitoring, 14-28 
user defined, 6-1, 6-57 
valid APIC interrupts, 10-14 
vectors, 6-1 
virtual-8086 mode, 19-6 
INTO instruction, 2-5, 3-8, 6-4, 6-29, 17-10 
INTR# pin, 6-2, 6-7 
Invalid opcode exception (HUD), 2-16, 6-31, 6-52, 12-1, 17-3, 21-5, 
21-11, 21-20, 21-21, 30-3 
Invalid TSS exception (HTS), 6-36, 7-6 
Invalid-operation exception, x87 FPU, 21-11, 21-14 
INVD instruction, 2-24, 5-24, 11-17, 21-4 
INVLPG instruction, 2-24, 5-24, 21-4, 24-3 
IOPL (I/O privilege level) field, EFLAGS register 
description of, 2-10 
on return from exception, interrupt handler, 6-13 
sensitive instructions in virtual-8086 mode, 19-10 
virtual interrupt, 2-11 
IPI (see interprocessor interrupt) 
IRET instruction, 3-8, 6-7, 6-13, 6-17, 6-21, 7-10, 7-11, 8-17, 19-5, 
19-19, 24-7 
IRETD instruction, 2-10, 8-17 
IRR 
Interrupt Request Register, 10-40, 10-42, 10-48 
IRR (interrupt request register), local APIC, 10-31 
ISR 
In Service Register, 10-39, 10-42, 10-48 
1/0 
breakpoint exception conditions, 17-9 
in virtual-8086 mode, 19-10 
instruction restart flag 
SMM revision identifier field, 30-15 
instruction restart flag, SMM revision identifier field, 30-15 
IO_SMI bit, 30-12 
I/O permission bit map, TSS, 7-5 
map base address field, TSS, 7-5 
restarting following SMI interrupt, 30-15 
saving I/O state, 30-12 
SMM state save map, 30-12 
1/0 APIC, 10-27 
bus arbitration, 10-27 
description of, 10-1 
external interrupts, 6-3 
information about, 10-1 
interrupt sources, 10-2 
local APIC and I/O APIC, 10-2, 10-3 
overview of, 10-1 
valid interrupts, 10-14 
See also: local APIC 


J 
JMP instruction, 2-5, 3-8, 5-10, 5-15, 7-2, 7-9, 7-11 


K 
KEN# pin, 11-13, 21-36 


L 


LO-L3 (local breakpoint enable) flags 
DR7 register, 17-4 
L1 (level 1) cache 
caching methods, 11-6 
CPUID feature flag, 11-18 
description of, 11-4 
effect of using write-through memory, 11-9 
introduction of, 21-30 
invalidating and flushing, 11-17 
MESI cache protocol, 11-9 
shared and adaptive mode, 11-18 
L2 (level 2) cache 
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caching methods, 11-6 
description of, 11-4 
disabling, 11-17 
effect of using write-through memory, 11-9 
introduction of, 21-30 
invalidating and flushing, 11-17 
MESI cache protocol, 11-9 
L3 (level 3) cache 
caching methods, 11-6 
description of, 11-4 
disabling and enabling, 11-13, 11-17 
effect of using write-through memory, 11-9 
introduction of, 21-31 
invalidating and flushing, 11-17 
MESI cache protocol, 11-9 
LAR instruction, 2-24, 5-24 
Larger page sizes 
introduction of, 21-31 
support for, 21-19 
Last branch 
interrupt & exception recording 
description of, 17-11, 17-25, 17-27, 17-30, 17-32, 17-34, 17-36, 
17-38, 17-39 
record stack, 17-16, 17-17, 17-25, 17-26, 17-33, 17-35, 17-37, 
17-39 
record top-of-stack pointer, 17-16, 17-26, 17-33, 17-37, 17-39 
LastBranchFromlP MSR, 17-40 
LastBranchTolP MSR, 17-40 
LastExceptionFromlP MSR, 17-36, 17-37, 17-40 
LastExceptionToIP MSR, 17-36, 17-37, 17-40 
LBR (last branch/interrupt/exception) flag, DEBUGCTLMSR MSR, 17-13, 
17-34, 17-39, 17-40 
LDR 
Logical Destination Register, 10-42, 10-46, 10-47 
LDS instruction, 3-8, 5-8 
LDT 
associated with a task, 7-3 
description of, 2-3, 2-5, 3-15 
index into with index field of segment selector, 3-7 
pointer to in TSS, 7-5 
pointers to exception and interrupt handlers, 6-12 
segment descriptors in, 3-9 
segment selector field, TSS, 7-16 
selecting with TI (table indicator) flag of segment selector, 3-7 
setting up during initialization, 9-10 
task switching, 7-9 
task-gate descriptor, 7-8 
use in address translation, 3-6 
LDTR register 
description of, 2-3, 2-5, 2-6, 2-12, 3-15 
IA-32e mode, 2-12 
limit, 5-5 
storing, 3-16 
LE (local exact breakpoint enable) flag, DR7 register, 17-4, 17-9 
LENO-LEN3 (Length) fields, DR7 register, 17-4, 17-5 
LES instruction, 3-8, 5-8, 6-31 
LFENCE instruction, 2-15, 8-6, 8-15, 8-16, 8-17 
LFS instruction, 3-8, 5-8 
LGDT instruction, 2-23, 5-23, 8-17, 9-10, 21-20 
LGS instruction, 3-8, 5-8 
LIDT instruction, 2-23, 5-24, 6-9, 8-17, 9-9, 19-5, 21-27 
Limit checking 
description of, 5-4 
pointer offsets are within limits, 5-25 
Limit field, segment descriptor, 5-2, 5-4 
Linear address 
description of, 3-6 
IA-32e mode, 3-7 
introduction to, 2-6 
Linear address space, 3-6 
defined, 3-1 
of task, 7-17 
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Link (to previous task) field, TSS, 6-17 automatic bus locking, 8-3 
Linking tasks bus locking, 8-3 
mechanism, 7-15 effects on caches, 8-5 
modifying task linkages, 7-16 loading a segment descriptor, 21-20 
LINT pins on IA-32 processors, 21-35 
function of, 6-2 overview of, 8-1 
LLDT instruction, 2-23, 5-23, 8-17 software-controlled bus locking, 8-3 
LMSW instruction, 2-23, 5-24, 24-3, 24-7 LOCK# signal, 2-25, 8-1, 8-3, 8-4, 8-5 
Local APIC, 10-39 Logical address 
64-bit mode, 10-33 description of, 3-6 
APIC_ID value, 8-34 IA-32e mode, 3-7 
arbitration over the APIC bus, 10-27 Logical address space, of task, 7-18 
arbitration over the system bus, 10-27 Logical destination mode, local APIC, 10-24 
block diagram, 10-4 Logical processors 
cluster model, 10-25 per physical package, 8-25 
CR8 usage, 10-33 Logical x2APIC ID, 10-47 
current-count register, 10-17 low-temperature interrupt enable bit, 14-36, 14-39 
description of, 10-1 LSL instruction, 2-24, 5-25 
detecting with CPUID, 10-8 LSS instruction, 3-8, 5-8 
DFR (destination format register), 10-25 LTR instruction, 2-23, 5-24, 7-7, 8-17, 9-11 
divide configuration register, 10-16 LVT (see Local vector table) 
enabling and disabling, 10-8 
external interrupts, 6-2 M 
features 
Pentium 4 and Intel Xeon, 21-28 Machine-check architecture 
Pentium and P6, 21-28 availability of MCA and exception, 15-19 
focus processor, 10-27 compatibility with Pentium processor, 15-1 
global enable flag, 10-8 compound error codes, 15-21 
IA32_APIC_BASE MSR, 10-8 CPUID flags, 15-19 
initial-count register, 10-16, 10-17 error codes, 15-20, 15-21 
internal error interrupts, 10-2 error-reporting bank registers, 15-2 
interrupt command register (ICR), 10-20 error-reporting MSRs, 15-5 
interrupt destination, 10-27 extended machine check state MSRs, 15-12 
interrupt distribution mechanism, 10-26 external bus errors, 15-27 
interrupt sources, 10-2 first introduced, 21-22 
IRR (interrupt request register), 10-31 global MSRs, 15-2 
I/O APIC, 10-1 initialization of, 15-19 
local APIC and 82489Dx, 21-28 introduction of in |A-32 processors, 21-36 
local APIC and I/O APIC, 10-2, 10-3 logging correctable errors, 15-29, 15-31, 15-35 
local vector table (LVT), 10-12 machine-check exception handler, 15-28 
logical destination mode, 10-24 machine-check exception (HMC), 15-1 
LVT (local-APIC version register), 10-11 MSRs, 15-2 
mapping of resources, 8-34 overview of MCA, 15-1 
MDA (message destination address), 10-24 Pentium processor exception handling, 15-29 
overview of, 10-1 Pentium processor style error reporting, 15-13 
performance-monitoring counter, 18-133 simple error codes, 15-21 
physical destination mode, 10-24 writing machine-check software, 15-27, 15-28 
receiving external interrupts, 6-2 Machine-check exception (#MC), 6-51, 15-1, 15-19, 15-28, 21-21, 21-36 
register address map, 10-6, 10-39 Mapping of shared resources, 8-34 
shared resources, 8-34 Maskable hardware interrupts 
SMI interrupt, 30-2 description of, 6-3 
spurious interrupt, 10-33 handling with virtual interrupt mechanism, 19-15 
spurious-interrupt vector register, 10-8 masking, 2-10, 6-7 — 
state after a software (INIT) reset, 10-11 MCA flag, CPUID instruction, 15-19 
state after INIT-deassert message, 10-11 MCE flag, CPUID instruction, 15-19 
state after power-up reset, 10-10 MCE (machine-check enable) flag 
state of, 10-34 CR4 control register, 2-17, 21-18 
SVR (spurious-interrupt vector register), 10-8 MDA (message destination address) 
timer, 10-16 local APIC, 10-24 
timer generated interrupts, 10-1 Memory, 11-1 
TMR (trigger mode register), 10-31 Memory management 
valid interrupts, 10-14 introduction to, 2-6 
version register, 10-11 overview, 3-1 
Local descriptor table register (see LDTR) paging, 3-1, 3-2 
Local descriptor table (see LDT) registers, 2-11 
Local vector table (LVT) segments, 3-1, 3-2, 3-7 
description of, 10-12 Memory ordering 
thermal entry, 14-31 in IA-32 processors, 21-34 
Local x2APIC, 10-32, 10-42, 10-47 overview, 8-5 _ 
Local xAPIC ID, 10-42 processor ordering, 8-5 
LOCK prefix, 2-25, 6-31, 8-1, 8-3, 8-15, 21-35 strengthening or weakening, 8-15 
Locked (atomic) operations write ordering, 8-5 
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Memory type range registers (see MTRRs) 
Memory types 
caching methods, defined, 11-6 
choosing, 11-8 
MTRR types, 11-21 
selecting for Pentium Ill and Pentium 4 processors, 11-15 
selecting for Pentium Pro and Pentium Il processors, 11-14 
UC (strong uncacheable), 11-6 
UC- (uncacheable), 11-6 
WB (write back), 11-7 
WC (write combining), 11-7 
WP (write protected), 11-7 
writing values across pages with different memory types, 11-16 
WT (write through), 11-7 
MemTypeGet() function, 11-29 
MemTypeSet() function, 11-31 
MESI cache protocol, 11-5, 11-9 
Message address register, 10-35 
Message data register format, 10-36 
Message signalled interrupts 
message address register, 10-35 
message data register format, 10-35 
MFENCE instruction, 2-15, 8-6, 8-15, 8-16, 8-17 
Microcode update facilities 
authenticating an update, 9-37 
BIOS responsibilities, 9-38 
calling program responsibilities, 9-39 
checksum, 9-33 
extended signature table, 9-31 
family OFH processors, 9-28 
field definitions, 9-28 
format of update, 9-28 
function OOH presence test, 9-42 
function 01H write microcode update data, 9-43 
function 02H microcode update control, 9-46 
function 03H read microcode update data, 9-47 
general description, 9-28 
HT Technology, 9-35 
INT 15H-based interface, 9-42 
overview, 9-27 
process description, 9-28 
processor identification, 9-32 
processor signature, 9-32 
return codes, 9-48 
update loader, 9-34 
update signature and verification, 9-36 
update specifications, 9-37 
VMX non-root operation, 24-11 
Mixing 16-bit and 32-bit code 
in IA-32 processors, 21-33 
overview, 20-1 
MMX technology 
debugging MMX code, 12-5 
effect of MMX instructions on pending x87 floating-point exceptions, 
12-5 
emulation of the MMX instruction set, 12-1 
exceptions that can occur when executing MMX instructions, 12-1 
introduction of into the IA-32 architecture, 21-2 
register aliasing, 12-1 
state, 12-1 
state, saving and restoring, 12-3 
system programming, 12-1 
task or context switches, 12-4 
using TS flag to control saving of MMX state, 13-7 
Mode switching 
example, 9-14 
real-address and protected mode, 9-13 
to SMM, 30-2 


Model and stepping information, following processor initialization or reset 


, 9-5 
Model-specific registers (see MSRs) 
Modes of operation (see Operating modes) 


INDEX 


MONITOR instruction, 24-3 
MOV instruction, 3-8, 5-8 
MOV (control registers) instructions, 2-23, 5-24, 8-17, 9-13 
MOV (debug registers) instructions, 2-24, 5-24, 8-17, 17-10 
MOVNTDQ instruction, 8-6, 11-17 
MOVNTI instruction, 2-15, 8-6, 11-17 
MOVNTPD instruction, 8-6, 11-17 
MOVNTPS instruction, 8-6, 11-17 
MOVNTQ instruction, 8-6, 11-17 
MP (monitor coprocessor) flag 

CRO control register, 2-15, 2-16, 6-32, 9-6, 9-7, 12-1, 21-8 
MSR 

Model Specific Register, 10-38, 10-39 
MSRs 

description of, 9-7 

introduction of in |A-32 processors, 21-36 

introduction to, 2-6 

machine-check architecture, 15-2 

reading and writing, 2-20, 2-21, 2-26 

reading & writing in 64-bit mode, 2-26 
MSR_DEBUBCTLB MSR, 17-12, 17-28, 17-37, 17-38 


MSR_DEBUGCTLA MSR, 17-11, 17-18, 17-23, 17-24, 17-33, 17-34, 18-5, 


18-9, 18-39, 18-50, 18-73, 18-78, 18-83, 18-98, 18-99 
MSR_DEBUGCTLB MSR, 17-12, 17-36, 17-38 
MSR_IFSB_CNTR7 MSR, 18-124 
MSR_IFSB_CTRL6 MSR, 18-124 
MSR_IFSB_DRDYO MSR, 18-123 
MSR_IFSB_DRDY1 MSR, 18-123 
MSR_IFSB_IBUSQO MSR, 18-123 
MSR_IFSB_IBUSQ1 MSR, 18-123 
MSR_IFSB_ISNPQO MSR, 18-123 
MSR_IFSB_ISNPQ1 MSR, 18-123 
MSR_LASTBRANCH_n MSR, 17-17, 17-35 
MSR_LASTBRANCH_n_FROM_IP MSR, 17-17, 17-35, 17-36 
MSR_LASTBRANCH_n_TO_IP MSR, 17-17, 17-35, 17-36 
MSR_LASTBRANCH_TOS MSR, 17-35 
MSR_LER_FROM_LIP MSR, 17-36, 17-37 
MSR_LER_TO_LIP MSR, 17-36, 17-37 
MTRR feature flag, CPUID instruction, 11-21 
MTRRcap MSR, 11-21 
MTRRfix MSR, 11-23 
MTRRs, 8-15 

base & mask calculations, 11-26, 11-27 

cache control, 11-13 

description of, 9-8, 11-20 

dual-core processors, 8-33 

enabling caching, 9-7 

feature identification, 11-21 

fixed-range registers, 11-23 

IA32_MTRRCAP MSR, 11-21 

IA32_MTRR_DEF_TYPE MSR, 11-22 

initialization of, 11-29 

introduction of in |A-32 processors, 21-36 

introduction to, 2-6 

large page size considerations, 11-33 

logical processors, 8-33 

mapping physical memory with, 11-21 

memory types and their properties, 11-21 

MemTypeGet() function, 11-29 

MemTypeSet() function, 11-31 

multiple-processor considerations, 11-32 

precedence of cache controls, 11-13 

precedences, 11-28 

programming interface, 11-29 

remapping memory types, 11-29 

state of following a hardware reset, 11-20 

variable-range registers, 11-23, 11-25 
Multi-core technology 

See multi-threading support 
Multiple-processor management 

bus locking, 8-3 

guaranteed atomic operations, 8-2 
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initialization 
MP protocol, 8-18 
procedure, 8-56 

local APIC, 10-1 

memory ordering, 8-5 

MP protocol, 8-18 

overview of, 8-1 

SMM considerations, 30-16 
Multiple-processor system 

local APIC and I/O APICs, Pentium 4, 10-3 

local APIC and I/O APIC, P6 family, 10-3 
Multisegment model, 3-4 
Multitasking 

initialization for, 9-10, 9-11 

initializing [A-32e mode, 9-11 

linking tasks, 7-15 

mechanism, description of, 7-2 

overview, 7-1 

setting up TSS, 9-10 

setting up TSS descriptor, 9-10 
Multi-threading support 

executing multiple threads, 8-26 

handling interrupts, 8-26 

logical processors per package, 8-25 

mapping resources, 8-34 

microcode updates, 8-33 

performance monitoring counters, 8-33 

programming considerations, 8-34 


See also: Hyper-Threading Technology and dual-core technology 


MWAIT instruction, 24-3 
power management extensions, 14-27 
MXCSR register, 6-52, 9-8, 13-6 


NaN, compatibility, |A-32 processors, 21-9 
NE (numeric error) flag 


CRO control register, 2-15, 6-47, 9-6, 9-7, 21-8, 21-18 


NEG instruction, 8-3 


NetBurst microarchitecture (see Intel NetBurst microarchitecture) 


NMI interrupt, 2-25, 10-3 
description of, 6-2 
handling during initialization, 9-9 
handling in SMM, 30-11 
handling multiple NMIs, 6-6 
masking, 21-27 
receiving when processor is shutdown, 6-34 
reference information, 6-27 
vector, 6-2 
NMI# pin, 6-2, 6-27 
Nominal CPI method, 18-136 
Nonconforming code segments 
accessing, 5-11 
C (conforming) flag, 5-11 
description of, 3-13 
Non-halted clockticks, 18-136 
setting up counters, 18-136 
Non-Halted CPI method, 18-136 
Nonmaskable interrupt (see NMI) 
Non-precise event-based sampling 
defined, 18-104 
used for at-retirement counting, 18-114 
writing an interrupt service routine for, 17-24 
Non-retirement events, 18-103 
Non-sleep clockticks, 18-136 
NOT instruction, 8-3 
Notation 
bit and byte order, 1-7 
conventions, 1-6 
exceptions, 1-9 
hexadecimal and binary numbers, 1-8 
Instructions 
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operands, 1-8 

reserved bits, 1-7 

segmented addressing, 1-8 
NT (nested task) flag 

EFLAGS register, 2-10, 7-10, 7-11, 7-15 
Null segment selector, checking for, 5-6 
Numeric overflow exception (#0), 21-10 
Numeric underflow exception (#U), 21-10 
NV (invert) flag, Perf€vtSelO MSR 

(P6 family processors), 18-5, 18-132 
NW (not write-through) flag 


CRO control register, 2-15, 9-7, 11-12, 11-13, 11-16, 11-32, 21-18, 


21-19, 21-30 
NXE bit, 5-30 


Oo 


Obsolete instructions, 21-5, 21-15 
OF flag, EFLAGS register, 6-29 
On die digital thermal sensor, 14-34 
relevant MSRs, 14-34 
sensor enumeration, 14-34 
On-Demand 
clock modulation enable bits, 14-32 
On-demand 
clock modulation duty cycle bits, 14-32 
On-die sensors, 14-28 
Opcodes 
undefined, 21-5 
Operands 
instruction, 1-8 
operand-size prefix, 20-1 
Operating modes 
64-bit mode, 2-7 
compatibility mode, 2-7 
IA-32e mode, 2-7, 2-8 
introduction to, 2-7 
protected mode, 2-7 
SMM (system management mode), 2-7 
transitions between, 2-8 
virtual-8086 mode, 2-8 
VMX operation 
enabling and entering, 22-3 
OR instruction, 8-3 
OS (operating system mode) flag 


Perf€vtSel0 and Perf€vtSell MSRs (P6 only), 18-4, 18-131 


OSFXSR (FXSAVE/FXRSTOR support) flag 
CR4 control register, 2-18, 9-8, 13-2 


OSXMMEXCPT (SIMD floating-point exception support) flag, CR4 control 


register, 2-18, 6-52, 9-8, 13-3 
OUT instruction, 8-15, 24-2 
Out-of-spec status bit, 14-35, 14-38 
Out-of-spec status log, 14-35, 14-38, 14-39 


OUTS/OUTSB/OUTSW/OUTSD instruction, 17-9, 24-2 


Overflow exception (HOF), 6-29 
Overheat interrupt enable bit, 14-36, 14-39 


P 


P (present) flag 
Page-directory entry, 6-44 
page-table entry, 6-44 
segment descriptor, 3-11 
P5_MC_ADDR MSR, 15-13, 15-29 
P5_MC_TYPE MSR, 15-13, 15-29 
P6 family processors 
compatibility with FP software, 21-7 
description of, 1-1 


last branch, interrupt, and exception recording, 17-39 


PAE paging 
feature flag, CR4 register, 2-17, 2-18 
flag, CR4 control register, 3-6, 21-18, 21-19 


Page attribute table (PAT) 
compatibility with earlier |A-32 processors, 11-36 
detecting support for, 11-34 
IA32_PAT MSR, 11-34 
introduction to, 11-33 
memory types that can be encoded with, 11-34 
MSR, 11-13 
precedence of cache controls, 11-14 
programming, 11-35 
selecting a memory type with, 11-35 
Page directories, 2-6 
Page directory 
base address (PDBR), 7-5 
introduction to, 2-6 
overview, 3-2 
setting up during initialization, 9-10 
Page directory pointers, 2-6 
Page frame (see Page) 
Page tables, 2-6 
introduction to, 2-6 
overview, 3-2 
setting up during initialization, 9-10 
Page-directory entries, 8-3, 11-5 
Page-fault exception (HPF), 4-51, 6-44, 21-21 
Pages 
disabling protection of, 5-1 
enabling protection of, 5-1 
introduction to, 2-6 
overview, 3-2 
PG flag, CRO control register, 5-1 
split, 21-15 
Page-table entries, 8-3, 11-5, 11-19 
Paging 
combining segment and page-level protection, 5-29 
combining with segmentation, 3-5 
defined, 3-1 
IA-32e mode, 2-6 
initializing, 9-10 
introduction to, 2-6 
large page size MTRR considerations, 11-33 
mapping segments to pages, 4-51 
page-fault exception, 6-44, 6-54, 6-55 
page-level protection, 5-2, 5-3, 5-27 
page-level protection flags, 5-28 
virtual-8086 tasks, 19-7 
Parameter 
passing, between 16- and 32-bit call gates, 20-6 
translation, between 16- and 32-bit code segments, 20-6 
PAUSE instruction, 2-15, 24-3 
PBi (performance monitoring/breakpoint pins) flags, DEBUGCTLMSR MSR, 
17-38, 17-40 
PC (pin control) flag, Perf€vtSel0 and Perf€vtSel1 MSRs (P6 family 
processors), 18-4, 18-132 
PCO and PC1 (pin control) fields, CESR MSR (Pentium processor), 18-134 
PCD pin (Pentium processor), 11-13 
PCD (page-level cache disable) flag 
CR3 control register, 2-17, 11-13, 21-18, 21-30 
page-directory entries, 9-7, 11-13, 11-33 
page-table entries, 9-7, 11-13, 11-33, 21-31 
PCE (performance monitoring counter enable) flag, CR4 control register, 
2-18, 5-24, 18-106, 18-132 
PCE (performance-monitoring counter enable) flag, CR4 control register, 
21-18 
PDBR (see CR3 control register) 
PE (protection enable) flag, CRO control register, 2-17, 5-1, 9-10, 9-13, 
30-9 
PEBS records, 17-21 
PEBS (precise event-based sampling) facilities 
availability of, 18-116 
description of, 18-104, 18-116 
DS save area, 17-18 
IA-32e mode, 17-21 
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PEBS buffer, 17-18, 18-116 
PEBS records, 17-18, 17-20 
writing a PEBS interrupt service routine, 18-117 
writing interrupt service routine, 17-24 
PEBS_UNAVAILABLE flag 
1A32_MISC_ENABLE MSR, 17-18 
Pentium 4 processor, 1-1 
compatibility with FP software, 21-7 
last branch, interrupt, and exception recording, 17-33 
time-stamp counter, 17-41 
Pentium II processor, 1-3 
Pentium Ill processor, 1-3 
Pentium M processor 
last branch, interrupt, and exception recording, 17-38 
time-stamp counter, 17-41 
Pentium Pro processor, 1-3 
Pentium processor, 1-1, 21-7 
compatibility with MCA, 15-1 
performance-monitoring counters, 18-133 
PerfCtrO and PerfCtr1 MSRs 
(P6 family processors), 18-131, 18-132 
Perf€vtSel0 and Perf€vtSell MSRs 
(P6 family processors), 18-131 
Perf€vtSel0 and Perf€vtSel1 MSRs (P6 family processors), 18-131 
Performance events 
architectural, 18-1 
Intel Core Solo and Intel Core Duo processors, 18-1 
non-architectural, 18-1 
Pentium 4 and Intel Xeon processors, 17-33 
Pentium M processors, 17-38 
Performance state, 14-1 
Performance-monitoring counters 
counted events (Pentium processors), 18-135 
description of, 18-1, 18-2 
interrupt, 10-1 
introduction of in |A-32 processors, 21-37 
monitoring counter overflow (P6 family processors), 18-133 
overflow, monitoring (P6 family processors), 18-133 
overview of, 2-7 
P6 family processors, 18-130 
Pentium Il processor, 18-130 
Pentium Pro processor, 18-130 
Pentium processor, 18-133 
reading, 2-25, 18-132 
setting up (P6 family processors), 18-131 
software drivers for, 18-132 
starting and stopping, 18-132 
PG (paging) flag 
CRO control register, 2-14, 5-1 
PG (paging) flag, CRO control register, 9-10, 9-13, 21-32, 30-9 
PGE (page global enable) flag, CR4 control register, 2-18, 11-13, 21-18, 
21-19 
PhysBase field, IA32_MTRR_PHYSBASEn MTRR, 11-24, 11-26 
Physical address extension 
introduction to, 3-6 
Physical address space 
4 GBytes, 3-6 
64 GBytes, 3-6 
addressing, 2-6 
defined, 3-1 
description of, 3-6 
IA-32e mode, 3-6 
mapped to a task, 7-17 
mapping with variable-range MTRRs, 11-23, 11-25 
Physical destination mode, local APIC, 10-24 
PhysMask 
IA32_MTRR_PHYSMASKn MTRR, 11-24, 11-26 
PMO/BPO and PM1/BP1 (performance-monitor) pins (Pentium processor), 
18-133, 18-134, 18-135 
PML4 tables, 2-6 
Pointers 
code-segment pointer size, 20-4 
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limit checking, 5-25 
validation, 5-24 
POP instruction, 3-8 
POPF instruction, 6-7, 17-10 
Power consumption 
software controlled clock, 14-28, 14-32 
Precise event-based sampling (see PEBS) 
PREFETCHh instruction, 2-15, 11-17 
Previous task link field, TSS, 7-5, 7-15, 7-16 
Privilege levels 
checking when accessing data segments, 5-8 
checking, for call gates, 5-15 
checking, when transferring program control between code segments, 
5-10 
description of, 5-6 
protection rings, 5-8 
Privileged instructions, 5-23 
Processor families 
O6H, 16-1 
OFH, 16-1 
Processor management 
initialization, 9-1 
local APIC, 10-1 
microcode update facilities, 9-27 
overview of, 8-1 
See also: multiple-processor management 
Processor ordering, description of, 8-5 
PROCHOT# log, 14-35, 14-38 
PROCHOT# or FORCEPR# event bit, 14-34, 14-38, 14-39 
Protected mode 
IDT initialization, 9-10 
initialization for, 9-9 
mixing 16-bit and 32-bit code modules, 20-1 
mode switching, 9-13 
PE flag, CRO register, 5-1 
switching to, 5-1, 9-13 
system data structures required during initialization, 9-9 
Protection 
combining segment & page-level, 5-29 
disabling, 5-1 
enabling, 5-1 
flags used for page-level protection, 5-2, 5-3 
flags used for segment-level protection, 5-2 
IA-32e mode, 5-3 
of exception, interrupt-handler procedures, 6-16 
overview of, 5-1 
page level, 5-1, 5-27, 5-28, 5-30 
page level, overriding, 5-29 
page-level protection flags, 5-28 
read/write, page level, 5-28 
segment level, 5-1 
user/supervisor type, 5-28 
Protection rings, 5-8 
PSE (page size extension) flag 
CR4 control register, 2-17, 11-20, 21-18, 21-19 
PSE-36 page size extension, 3-6 
Pseudo-functions 
VMfail, 29-2 
VMfaillnvalid, 29-2 
VMfailValid, 29-2 
VMsucceed, 29-2 
Pseudo-infinity, 21-9 
Pseudo-NaN, 21-9 
Pseudo-zero, 21-9 
P-state, 14-1 
PUSH instruction, 21-7 
PUSHF instruction, 6-7, 21-7 
PVI (protected-mode virtual interrupts) flag 
CR4 control register, 2-11, 2-17, 21-18 
PWT pin (Pentium processor), 11-13 
PWT (page-level write-through) flag 
CR3 control register, 2-17, 11-13, 21-18, 21-30 
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page-directory entries, 9-7, 11-13, 11-33 
page-table entries, 9-7, 11-33, 21-31 


Q 


QNaN, compatibility, IA-32 processors, 21-9 


R 
RDMSR instruction, 2-20, 2-21, 2-26, 5-24, 17-35, 17-40, 17-42, 
18-106, 18-131, 18-132, 18-133, 21-5, 21-36, 24-4, 24-9 
RDPMC instruction, 2-25, 5-24, 18-105, 18-131, 18-132, 21-4, 21-18, 
21-37, 24-4 
in 64-bit mode, 2-26 
RDTSC instruction, 2-25, 5-24, 17-42, 21-5, 24-4, 24-9 
in 64-bit mode, 2-26 
reading sensors, 14-34 
Read/write 
protection, page level, 5-28 
rights, checking, 5-25 
Real-address mode 
8086 emulation, 19-1 
address translation in, 19-2 
description of, 19-1 
exceptions and interrupts, 19-6 
IDT initialization, 9-8 
IDT, changing base and limit of, 19-5 
IDT, structure of, 19-5 
IDT, use of, 19-4 
initialization, 9-8 
instructions supported, 19-3 
interrupt and exception handling, 19-4 
interrupts, 19-6 
introduction to, 2-7 
mode switching, 9-13 
native 16-bit mode, 20-1 
overview of, 19-1 
registers supported, 19-3 
switching to, 9-14 
Recursive task switching, 7-16 
Related literature, 1-10 
Requested privilege level (see RPL) 
Reserved bits, 1-7, 21-2 
RESET# pin, 6-3, 21-16 
RESET# signal, 2-25 
Resolution in degrees, 14-36 
Restarting program or task, following an exception or interrupt, 6-5 
Restricting addressable domain, 5-28 
RET instruction, 5-10, 5-20, 20-6 
Returning 
from a called procedure, 5-20 
from an interrupt or exception handler, 6-13 
RF (resume) flag 
EFLAGS register, 2-10, 6-7 
RPL 
description of, 3-8, 5-8 
field, segment selector, 5-2 
RSM instruction, 2-25, 8-17, 21-5, 24-4, 30-1, 30-2, 30-3, 30-13, 30-15, 
30-18 
RsvdZ, 10-41 
R/S# pin, 6-3 
R/W (read/write) flag 
Page-directory entry, 5-1, 5-2, 5-28 
page-table entry, 5-1, 5-2, 5-28 
R/WO-R/W3 (read/write) fields 
DR7 register, 17-4, 21-20 


S 
S (descriptor type) flag 
segment descriptor, 3-11, 3-12, 5-2, 5-5 
SBB instruction, 8-3 
Segment descriptors 


access rights, 5-24 

access rights, invalid values, 21-19 

automatic bus locking while updating, 8-3 

base address fields, 3-10 

code type, 5-2 

data type, 5-2 

description of, 2-4, 3-9 

DPL (descriptor privilege level) field, 3-11, 5-2 

D/B (default operation size/default stack pointer size and/or upper 

bound) flag, 3-11, 5-4 

E (expansion direction) flag, 5-2, 5-4 

G (granularity) flag, 3-11, 5-2, 5-4 

limit field, 5-2, 5-4 

loading, 21-20 

P (segment-present) flag, 3-11 

S (descriptor type) flag, 3-11, 3-12, 5-2, 5-5 

segment limit field, 3-10 

system type, 5-2 

tables, 3-14 

TSS descriptor, 7-5, 7-6 

type field, 3-10, 3-12, 5-2, 5-5 

type field, encoding, 3-14 

when P (segment-present) flag is clear, 3-11 
Segment limit 

checking, 2-24 

field, segment descriptor, 3-10 
Segment not present exception (#NP), 3-11 
Segment registers 

description of, 3-8 

IA-32e mode, 3-9 

saved in TSS, 7-4 
Segment selectors 

description of, 3-7 

index field, 3-7 

null, 5-6 

null in 64-bit mode, 5-6 

RPL field, 3-8, 5-2 

TI (table indicator) flag, 3-7 
Segmented addressing, 1-8 
Segment-not-present exception (#NP), 6-38 
Segments 

64-bit mode, 3-5 

basic flat model, 3-3 

code type, 3-12 

combining segment, page-level protection, 5-29 

combining with paging, 3-5 

compatibility mode, 3-5 

data type, 3-12 

defined, 3-1 

disabling protection of, 5-1 

enabling protection of, 5-1 

mapping to pages, 4-51 

multisegment usage model, 3-4 

protected flat model, 3-3 

segment-level protection, 5-2, 5-3 

segment-not-present exception, 6-38 

system, 2-4 

types, checking access rights, 5-24 

typing, 5-5 

using, 3-2 

wraparound, 21-34 
SELF IPI register, 10-39 
Self-modifying code, effect on caches, 11-18 
Serializing, 8-17 
Serializing instructions 

CPUID, 8-17 

HT technology, 8-30 

non-privileged, 8-17 

privileged, 8-17 
SF (stack fault) flag, x87 FPU status word, 21-8 
SFENCE instruction, 2-15, 8-6, 8-15, 8-16, 8-17 
SGDT instruction, 2-23, 3-15 


Shared resources 
mapping of, 8-34 
Shutdown 
resulting from double fault, 6-34 
resulting from out of IDT limit condition, 6-34 
SIDT instruction, 2-23, 3-16, 6-9 
SIMD floating-point exception (#XM), 2-18, 6-52, 9-8 
SIMD floating-point exceptions 
description of, 6-52, 13-5 
handler, 13-3 
support for, 2-18 
Single-stepping 
breakpoint exception condition, 17-10 
on branches, 17-13 
on exceptions, 17-13 
on interrupts, 17-13 
TF (trap) flag, EFLAGS register, 17-10 
SLDT instruction, 2-23 
SLTR instruction, 3-16 
SMBASE 
default value, 30-4 
relocation of, 30-14 
SMI handler 
description of, 30-1 
execution environment for, 30-9 
exiting from, 30-3 
VMX treatment of, 30-16 
SMI interrupt, 2-25, 10-3 
description of, 30-1, 30-2 
IO_SMI bit, 30-12 
priority, 30-3 
switching to SMM, 30-2 
synchronous and asynchronous, 30-12 
VMX treatment of, 30-16 
SMI# pin, 6-3, 30-2, 30-15 
SMM 
asynchronous SMI, 30-12 
auto halt restart, 30-14 
executing the HLT instruction in, 30-14 
exiting from, 30-3 
handling exceptions and interrupts, 30-11 
introduction to, 2-7 
I/O instruction restart, 30-15 
I/O state implementation, 30-12 
native 16-bit mode, 20-1 
overview of, 30-1 
revision identifier, 30-13 
revision identifier field, 30-13 
switching to, 30-2 
switching to from other operating modes, 30-2 
synchronous SMI, 30-12 
VMX operation 
default RSM treatment, 30-18 
default SMI delivery, 30-17 
dual-monitor treatment, 30-19 
overview, 30-2 
protecting CR4.VMXE, 30-19 
RSM instruction, 30-18 
SMM monitor, 30-2 
SMM VM exits, 26-1, 30-19 
SMM-transfer VMCS, 30-19 
SMM-transfer VMCS pointer, 30-19 
VMCS pointer preservation, 30-17 
VMxX-critical state, 30-17 
SMRAM 
caching, 30-8 
state save map, 30-4 
structure of, 30-4 
SMSW instruction, 2-23, 24-9 
SNaN, compatibility, [A-32 processors, 21-9, 21-14 
Snooping mechanism, 11-6 
Software controlled clock 
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modulation control bits, 14-32 
power consumption, 14-28, 14-32 
Software interrupts, 6-4 
Software-controlled bus locking, 8-3 
Split pages, 21-15 
Spurious interrupt, local APIC, 10-33 
SSE extensions 
checking for with CPUID, 13-2 
checking support for FXSAVE/FXRSTOR, 13-2 
CPUID feature flag, 9-8 
EM flag, 2-16 
emulation of, 13-6 
facilities for automatic saving of state, 13-6, 13-7 
initialization, 9-8 
introduction of into the IA-32 architecture, 21-3 
providing exception handlers for, 13-4, 13-5 
providing operating system support for, 13-1 
saving and restoring state, 13-6 
saving state on task, context switches, 13-6 
SIMD Floating-point exception (#XM), 6-52 
using TS flag to control saving of state, 13-7 
SSE feature flag 
CPUID instruction, 13-2 
SSE2 extensions 
checking for with CPUID, 13-2 
checking support for FXSAVE/FXRSTOR, 13-2 
CPUID feature flag, 9-8 
EM flag, 2-16 
emulation of, 13-6 
facilities for automatic saving of state, 13-6, 13-7 
initialization, 9-8 
introduction of into the IA-32 architecture, 21-3 
providing exception handlers for, 13-4, 13-5 
providing operating system support for, 13-1 
saving and restoring state, 13-6 
saving state on task, context switches, 13-6 
SIMD Floating-point exception (#XM), 6-52 
using TS flag to control saving state, 13-7 
SSE2 feature flag 
CPUID instruction, 13-2 
SSE3 extensions 
checking for with CPUID, 13-2 
CPUID feature flag, 9-8 
EM flag, 2-16 
emulation of, 13-6 
example verifying SS3 support, 8-46, 8-50, 14-2 
facilities for automatic saving of state, 13-6, 13-7 
initialization, 9-8 
introduction of into the IA-32 architecture, 21-3 
providing exception handlers for, 13-4, 13-5 
providing operating system support for, 13-1 
saving and restoring state, 13-6 
saving state on task, context switches, 13-6 
using TS flag to control saving of state, 13-7 
SSE3 feature flag 
CPUID instruction, 13-2 
Stack fault exception (#SS), 6-40 
Stack fault, x87 FPU, 21-8, 21-13 
Stack pointers 
privilege level 0, 1, and 2 stacks, 7-5 
size of, 3-11 
Stack segments 
paging of, 2-6 
privilege level check when loading SS register, 5-10 
size of stack pointer, 3-11 
Stack switching 
exceptions/interrupts when switching stacks, 6-8 
IA-32e mode, 6-21 
inter-privilege level calls, 5-17 
Stack-fault exception (#SS), 21-34 
Stacks 
error code pushes, 21-32 


INDEX-16 Vol. 3D 


faults, 6-40 
for privilege levels 0, 1, and 2, 5-17 
interlevel RET/IRET 
from a 16-bit interrupt or call gate, 21-33 
interrupt stack table, 64-bit mode, 6-22 
management of control transfers for 
16- and 32-bit procedure calls, 20-4 
operation on pushes and pops, 21-32 
pointers to in TSS, 7-5 
stack switching, 5-17, 6-21 
usage on call to exception 
or interrupt handler, 21-33 
Stepping information, following processor initialization or reset, 9-5 
STl instruction, 6-7 
Store buffer 
caching terminology, 11-5 
characteristics of, 11-4 
description of, 11-5, 11-20 
in IA-32 processors, 21-34 
location of, 11-1 
operation of, 11-20 
STPCLK# pin, 6-3 
STR instruction, 2-23, 3-16, 7-7 
Strong uncached (UC) memory type 
description of, 11-6 
effect on memory ordering, 8-16 
use of, 9-8, 11-8 
Sub C-state, 14-27 
SUB instruction, 8-3 
Supervisor mode 
description of, 5-28 
U/S (user/supervisor) flag, 5-28 
SVR (spurious-interrupt vector register), local APIC, 10-8, 21-28 
SWAPGS instruction, 2-7 
SYSCALL instruction, 2-7, 5-22 
SYSENTER instruction, 3-8, 5-10, 5-20, 5-21 
SYSENTER_CS_MSR, 5-21 
SYSENTER_EIP_MSR, 5-21 
SYSENTER_ESP_MSR, 5-21 
SYSEXIT instruction, 3-8, 5-10, 5-20, 5-21 
SYSRET instruction, 2-7, 5-22 
System 
architecture, 2-1, 2-2 
data structures, 2-2 
instructions, 2-7, 2-22 
registers in |A-32e mode, 2-7 
registers, introduction to, 2-6 
segment descriptor, layout of, 5-2 
segments, paging of, 2-6 
System programming 
MMX technology, 12-1 
System-management mode (see SMM) 


T 
T (debug trap) flag, TSS, 7-5 
Task gates 
descriptor, 7-8 
executing a task, 7-2 
handling a virtual-8086 mode interrupt or exception through, 19-14 
IA-32e mode, 2-5 
in IDT, 6-10 
introduction for I|A-32e, 2-4 
introduction to, 2-4, 2-5 
layout of, 6-10 
referencing of TSS descriptor, 6-17 
Task management, 7-1 
data structures, 7-3 
mechanism, description of, 7-2 
Task register, 3-16 
description of, 2-13, 7-1, 7-7 
IA-32e mode, 2-13 


initializing, 9-11 
introduction to, 2-6 
Task switching 
description of, 7-3 
exception condition, 17-10 
operation, 7-10 
preventing recursive task switching, 7-16 
saving MMX state on, 12-4 
saving SSE/SSE2/SSE3 state 
on task or context switches, 13-6 
T (debug trap) flag, 7-5 
Tasks 
address space, 7-16 
description of, 7-1 
exception-handler task, 6-11 
executing, 7-2 
Intel 286 processor tasks, 21-37 
interrupt-handler task, 6-11 
interrupts and exceptions, 6-17 
linking, 7-15 
logical address space, 7-18 
management, 7-1 
mapping linear and physical address space, 7-17 
restart following an exception or interrupt, 6-5 
state (context), 7-2, 7-3 
structure, 7-1 
switching, 7-3 
task management data structures, 7-3 


TF (trap) flag, EFLAGS register, 2-9, 6-17, 17-10, 17-12, 17-34, 17-36, 
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Thermal monitoring 

advanced power management, 14-27 

automatic, 14-29 

automatic thermal monitoring, 14-28 

catastrophic shutdown detector, 14-28, 14-29 

clock-modulation bits, 14-32 

C-state, 14-27 

detection of facilities, 14-34 

Enhanced Intel SpeedStep Technology, 14-1 

IA32_APERF MSR, 14-2 

IA32_MPERF MSR, 14-1 

IA32_THERM_INTERRUPT MSR, 14-34 

IA32_THERM_STATUS MSR, 14-34 

interrupt enable/disable flags, 14-31 

interrupt mechanisms, 14-28 

MWAIT extensions for, 14-27 

on die sensors, 14-28, 14-34 

overview of, 14-1, 14-28 

performance state transitions, 14-30 

sensor interrupt, 10-1 

setting thermal thresholds, 14-34 

software controlled clock modulation, 14-28, 14-32 

status flags, 14-31 

status information, 14-31, 14-32 

stop clock mechanism, 14-28 

thermal monitor 1 (TM1), 14-29 

thermal monitor 2 (TM2), 14-29 

TM flag, CPUID instruction, 14-34 
Thermal status bit, 14-34, 14-38 
Thermal status log bit, 14-34, 14-38 
Thermal threshold #1 log, 14-35, 14-38, 14-39 
Thermal threshold #1 status, 14-35, 14-38 
Thermal threshold #2 log, 14-35, 14-38 
Thermal threshold #2 status, 14-35, 14-38, 14-39 
THERMTRIP# interrupt enable bit, 14-36, 14-39 
thread timeout indicator, 16-3, 16-7, 16-10, 16-13, 16-15 
Threshold #1 interrupt enable bit, 14-37, 14-39 
Threshold #1 value, 14-36, 14-39 
Threshold #2 interrupt enable, 14-37, 14-40 
Threshold #2 value, 14-37, 14-39 
TI (table indicator) flag, segment selector, 3-7 
Timer, local APIC, 10-16 


Time-stamp counter 
counting clockticks, 18-136 
description of, 17-41 
1A32_TIME_STAMP_COUNTER MSR, 17-41 
RDTSC instruction, 17-41 
reading, 2-25 
software drivers for, 18-132 
TSC flag, 17-41 
TSD flag, 17-41 
TLBs 
description of, 11-1, 11-5 
flushing, 11-19 
invalidating (flushing), 2-24 
relationship to PGE flag, 21-19 
relationship to PSE flag, 11-20 
T1 and TM2 
See: thermal monitoring, 14-29 
TMR 
Trigger Mode Register, 10-32, 10-40, 10-42, 10-48 
TMR (Trigger Mode Register), local APIC, 10-31 
TPR 
Task Priority Register, 10-39, 10-42 
TR (trace message enable) flag 


DEBUGCTLMSR MSR, 17-12, 17-34, 17-37, 17-38, 17-40 


Trace cache, 11-4, 11-5 
Transcendental instruction accuracy, 21-8, 21-14 
Translation lookaside buffer (see TLB) 
Trap gates 
difference between interrupt and trap gates, 6-17 
for 16-bit and 32-bit code modules, 20-1 
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handling a virtual-8086 mode interrupt or exception through, 19-12 


in IDT, 6-10 

introduction for I|A-32e, 2-4 

introduction to, 2-4, 2-5 

layout of, 6-10 
Traps 

description of, 6-5 

restarting a program or task after, 6-5 
TS (task switched) flag 

CRO control register, 2-15, 2-23, 6-32, 12-1, 13-3, 13-7 
TSD (time-stamp counter disable) flag 

CR4 control register, 2-17, 5-24, 17-42, 21-18 
TSS 

16-bit TSS, structure of, 7-18 

32-bit TSS, structure of, 7-3 

64-bit mode, 7-19 

CR3 control register (PDBR), 7-4, 7-17 

description of, 2-4, 2-5, 7-1, 7-3 

EFLAGS register, 7-4 

EFLAGS.NT, 7-15 

EIP, 7-5 

executing a task, 7-2 

floating-point save area, 21-12 

format in 64-bit mode, 7-19 

general-purpose registers, 7-4 

IA-32e mode, 2-5 

initialization for multitasking, 9-10 

interrupt stack table, 7-19 

invalid TSS exception, 6-36 

IRET instruction, 7-15 

I/O map base address field, 7-5, 21-29 

I/O permission bit map, 7-5, 7-19 

LDT segment selector field, 7-5, 7-16 

link field, 6-17 

order of reads/writes to, 21-29 

pointed to by task-gate descriptor, 7-8 

previous task link field, 7-5, 7-15, 7-16 

privilege-level 0, 1, and 2 stacks, 5-17 

referenced by task gate, 6-17 

segment registers, 7-4 

T (debug trap) flag, 7-5 

task register, 7-7 
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using 16-bit TSSs in a 32-bit environment, 21-29 
virtual-mode extensions, 21-29 
TSS descriptor 
B (busy) flag, 7-5 
busy flag, 7-16 
initialization for multitasking, 9-10 
structure of, 7-5, 7-6 
TSS segment selector 
field, task-gate descriptor, 7-8 


writes, 21-29 
Type 
checking, 5-5 


field, IA32_MTRR_DEF_TYPE MSR, 11-22 

field, IA32_MTRR_PHYSBASEn MTRR, 11-24, 11-26 
field, segment descriptor, 3-10, 3-12, 3-14, 5-2, 5-5 
of segment, 5-5 


U 


UC- (uncacheable) memory type, 11-6 
UD2 instruction, 21-4 
Uncached (UC-) memory type, 11-8 
Uncached (UC) memory type (see Strong uncached (UC) memory type) 
Undefined opcodes, 21-5 
Unit mask field, Perf€vtSelO and Perf€vtSel1 MSRs (P6 family processors) 
, 18-4, 18-8, 18-10, 18-11, 18-12, 18-14, 18-25, 18-27, 
18-35, 18-36, 18-54, 18-56, 18-57, 18-96, 18-97, 18-98, 
18-131, 18-140, 18-141, 18-142, 18-143, 18-146 
Un-normal number, 21-9 
User mode 
description of, 5-28 
U/S (user/supervisor) flag, 5-28 
User-defined interrupts, 6-1, 6-57 
USR (user mode) flag, Perf€vtSelO and PerfE€vtSel1 MSRs (P6 family 
processors), 18-4, 18-8, 18-10, 18-11, 18-12, 18-25, 18-27, 
18-35, 18-36, 18-54, 18-56, 18-57, 18-96, 18-97, 18-98, 
18-131, 18-140, 18-141, 18-142, 18-143, 18-146 
U/S (user/supervisor) flag 
page-directory entry, 5-1, 5-2, 5-28 
page-table entries, 19-8 
page-table entry, 5-1, 5-2, 5-28 


Vv 
V (valid) flag 
IA32_MTRR_PHYSMASKn MTRR, 11-24, 11-26 
Variable-range MTRRs, description of, 11-23, 11-25 
VCNT (variable range registers count) field, |A32_MTRRCAP MSR, 11-22 
Vectors 
exceptions, 6-1 
interrupts, 6-1 
VERR instruction, 2-24, 5-25 
VERW instruction, 2-24, 5-25 
VIF (virtual interrupt) flag 
EFLAGS register, 2-11, 21-6, 21-7 
VIP (virtual interrupt pending) flag 
EFLAGS register, 2-11, 21-6, 21-7 
Virtual memory, 2-6, 3-1, 3-2 
Virtual-8086 mode 
8086 emulation, 19-1 
description of, 19-5 
emulating 8086 operating system calls, 19-18 
enabling, 19-6 
entering, 19-8 
exception and interrupt handling overview, 19-11 
exceptions and interrupts, handling through a task gate, 19-14 
exceptions and interrupts, handling through a trap or interrupt gate, 
19-12 
handling exceptions and interrupts through a task gate, 19-14 
interrupts, 19-6 
introduction to, 2-8 
IOPL sensitive instructions, 19-10 
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\/O-port-mapped I/O, 19-11 
leaving, 19-9 
memory mapped I/O, 19-11 
native 16-bit mode, 20-1 
overview of, 19-1 
paging of virtual-8086 tasks, 19-7 
protection within a virtual-8086 task, 19-8 
special I/O buffers, 19-11 
structure of a virtual-8086 task, 19-7 
virtual I/O, 19-10 
VM flag, EFLAGS register, 2-10 
Virtual-8086 tasks 
paging of, 19-7 
protection within, 19-8 
structure of, 19-7 
VM entries 
basic VM-entry checks, 25-2 
checking guest state 
control registers, 25-9 
debug registers, 25-9 
descriptor-table registers, 25-12 
MSRs, 25-9 
non-register state, 25-13 
RIP and RFLAGS, 25-12 
segment registers, 25-10 
checks on controls, host-state area, 25-2 
registers and MSRs, 25-7 
segment and descriptor-table registers, 25-8 
VMX control checks, 25-2 
exit-reason numbers, C-1 
loading guest state, 25-15 
control and debug registers, MSRs, 25-16 
RIP, RSP, RFLAGS, 25-17 
segment & descriptor-table registers, 25-17 
loading MSRs, 25-18 
failure cases, 25-18 
VM-entry MSR-load area, 25-18 
overview of failure conditions, 25-1 
overview of steps, 25-1 
VMLAUNCH and VMRESUME, 25-1 
See also: VMCS, VMM, VM exits 
VM exits 
architectural state 
existing before exit, 26-1 
updating state before exit, 26-1 
basic VM-exit information fields, 26-4 
basic exit reasons, 26-4 
exit qualification, 26-4 
exception bitmap, 26-1 
exceptions (faults, traps, and aborts), 24-5 
exit-reason numbers, C-1 
external interrupts, 24-6 
IA-32 faults and VM exits, 24-1 
INITs, 24-6 
instructions that cause: 
conditional exits, 24-2 
unconditional exits, 24-2 
interrupt-window exiting, 24-6 
non-maskable interrupts (NMIs), 24-6 
page faults, 24-5 
start-up IPIs (SIPIs), 24-6 
task switches, 24-6 
See also: VMCS, VMM, VM entries 
VM (virtual-8086 mode) flag 
EFLAGS register, 2-8, 2-10 
VMCALL instruction, 29-1 
VMCLEAR instruction, 29-1 
VMCS 
error numbers, 29-31 
field encodings, 1-6, B-1 
16-bit guest-state fields, B-1 
16-bit host-state fields, B-2 


32-bit control fields, B-1, B-6 
32-bit guest-state fields, B-7 
32-bit read-only data fields, B-7 
64-bit control fields, B-2 
64-bit guest-state fields, B-5, B-6 
natural-width control fields, B-8 
natural-width guest-state fields, B-9 
natural-width host-state fields, B-10 
natural-width read-only data fields, B-9 
format of VMCS region, 23-2 
guest-state area, 23-3 
guest non-register state, 23-6 
guest register state, 23-4 
host-state area, 23-3, 23-8 
introduction, 23-1 
migrating between processors, 23-26 
software access to, 23-26 
VMCS data, 23-2, 24-19 
VMCS pointer, 23-1 
VMCS region, 23-1 
VMCS revision identifier, 23-2, 24-19 
VM-entry control fields, 23-3, 23-20 
entry controls, 23-21 
entry controls for event injection, 23-22 
entry controls for MSRs, 23-21 
VM-execution control fields, 23-3, 23-9 
controls for CR8 accesses, 23-14 
CR3-target controls, 23-14 
exception bitmap, 23-13 
I/O bitmaps, 23-13 
masks & read shadows CRO & CR4, 23-14 
pin-based controls, 23-9 
processor-based controls, 23-10 
time-stamp counter offset, 23-13 
VM-exit control fields, 23-3, 23-19 
exit controls, 23-19 
exit controls for MSRs, 23-20 
VM-exit information fields, 23-3, 23-23 
basic exit information, 23-23, C-1 
basic VM-exit information, 23-23 
exits due to instruction execution, 23-25 
exits due to vectored events, 23-24 
exits occurring during event delivery, 23-25 
VM-instruction error field, 23-26 
VM-instruction error field, 25-1, 29-31 
VMREAD instruction 
field encodings, 1-6, B-1 
VMWRITE instruction 
field encodings, 1-6, B-1 
VMx-abort indicator, 23-2, 24-19 
See also: VM entries, VM exits, VMM, VMX 
VME (virtual-8086 mode extensions) flag, CR4 control register, 2-11, 
2-17, 21-18 
VMLAUNCH instruction, 29-1 
VMM 
VM exits, 26-1 
See also: VMCS, VM entries, VM exits, VMX 
VMPTRLD instruction, 29-1 
VMPTRST instruction, 29-1 
VMREAD instruction, 29-1 
field encodings, B-1 
VMRESUME instruction, 29-1 
VMWRITE instruction, 29-1 
field encodings, B-1 
VMX 
A20M# signal, 22-4 
capability MSRs 
overview, 22-2, A-1 
IA32_VMX_BASIC MSR, 23-3, A-1, A-2 
IA32_VMX_CRO_FIXEDO MSR, A-6 
IA32_VMX_CRO_FIXED1 MSR, A-6 
IA32_VMX_ENTRY_CTLS MSR, A-2, A-5 
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IA32_VMX_EXIT_CTLS MSR, A-2, A-4, A-5 
IA32_VMX_MISC MSR, 23-6, 25-3, 25-13, 30-26, A-6 
IA32_VMX_PINBASED_CTLS MSR, A-2, A-3 
IA32_VMX_PROCBASED_CTLS MSR, 23-10, A-2, A-3, A-4, A-8 
CPUID instruction, 22-2, A-1 
CR4 control register, 22-3 
CR4 fixed bits, A-7 
entering operation, 22-3 
guest software, 22-1 
IA32_FEATURE_CONTROL MSR, 22-3 
INIT# signal, 22-4 
instruction set, 22-2 
introduction, 22-1 
microcode update facilities, 24-11 
non-root operation, 22-1 
event blocking, 24-12 
instruction changes, 24-7 
overview, 24-1 
task switches not allowed, 24-12 
see VM exits 
operation restrictions, 22-3 
root operation, 22-1 
SMM 
CR4.VMXE reserved, 30-19 
overview, 30-2 
RSM instruction, 30-18 
VMCS pointer, 30-17 
VMxX-critical state, 30-17 
testing for support, 22-2 
virtual-machine control structure (VMCS), 22-2 
virtual-machine monitor (VMM), 22-1 
VM entries and exits, 22-1 
VM exits, 26-1 
VMCS pointer, 22-2 
VMM life cycle, 22-2 
VMXOFF instruction, 22-3 
VMXON instruction, 22-3 
VMXON pointer, 22-3 
VMXON region, 22-3 
See also: VMM, VMCS, VM entries, VM exits 
VMXOFF instruction, 22-3, 29-1 
VMXON instruction, 22-3, 29-1 
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WAIT/FWAIT instructions, 6-32, 21-8, 21-15, 21-16 
WB (write back) memory type, 8-16, 11-7, 11-8 
WB (write-back) pin (Pentium processor), 11-13 
WBINVD instruction, 2-24, 5-24, 11-16, 11-17, 21-4 
WB/WT# pins, 11-13 
WC buffer (see Write combining (WC) buffer) 
WC (write combining) 
flag, [A32_MTRRCAP MSR, 11-22 
memory type, 11-7, 11-8 
WP (write protected) memory type, 11-7 
WP (write protect) flag 
CRO control register, 2-15, 5-28, 21-18 
Write 
hit, 11-5 
Write combining (WC) buffer, 11-4, 11-8 
Write-back caching, 11-6 
WRMSR instruction, 2-20, 2-21, 2-26, 5-24, 8-17, 17-34, 17-39, 17-42, 
18-106, 18-131, 18-132, 18-133, 21-5, 21-36 
WT (write through) memory type, 11-7, 11-8 
WT# (write-through) pin (Pentium processor), 11-13 


Xx 
x2APIC ID, 10-41, 10-42, 10-45, 10-47 
x2APIC Mode, 10-32, 10-38, 10-39, 10-41, 10-42, 10-45, 10-46, 10-47 
x87 FPU 
compatibility with [A-32 x87 FPUs and math coprocessors, 21-7 
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configuring the x87 FPU environment, 9-6 
device-not-available exception, 6-32 
effect of MMX instructions on pending x87 floating-point exceptions, 
12-5 
effects of MMX instructions on x87 FPU state, 12-3 
effects of MMX, x87 FPU, FXSAVE, and FXRSTOR instructions on x87 
FPU tag word, 12-3 
error signals, 21-11 
initialization, 9-5 
instruction synchronization, 21-15 
register stack, aliasing with MMX registers, 12-2 
setting up for software emulation of x87 FPU functions, 9-6 
using TS flag to control saving of x87 FPU state, 13-7 
x87 floating-point error exception (#MF), 6-47 
x87 FPU control word 
compatibility, IA-32 processors, 21-9 
x87 FPU floating-point error exception (#MF), 6-47 
x87 FPU status word 
condition code flags, 21-8 
x87 FPU tag word, 21-9 
XADD instruction, 8-3, 21-4 
xAPIC, 10-39, 10-41 
determining lowest priority processor, 10-26 
interrupt control register, 10-22 
introduction to, 10-4 
message passing protocol on system bus, 10-34 
new features, 21-28 
spurious vector, 10-33 
using system bus, 10-4 
xAPIC Mode, 10-32, 10-38, 10-42, 10-45, 10-47 
XCHG instruction, 8-3, 8-16 
XCRO, 2-19 
XGETBV, 2-19, 2-23 
XMM registers, saving, 13-6 
XOR instruction, 8-3 
XSAVE, 2-19, 13-7, 13-8, 13-9, 13-10 
XSETBV, 2-19, 2-20, 2-23, 2-26 


Zz 
ZF flag, EFLAGS register, 5-25 
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